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Executive Summary 


This final report presents findings from a multi-year evaluation of the Comprehensive 
Technical Assistance Centers, a federally funded program that provides technical assistance to 
states in connection with the Elementary and Secondary Education Act, as reauthorized by the 
No Child Left Behind (NCLB) Act of 2001. The law authorizing the Centers, the Educational 
Technical Assistance Act of 2002, mandated that a national evaluation of the program be 
conducted by the Institute of Education Sciences (IES). The legislation indicated that the 
evaluation should “include an analysis of the services provided... [and] the extent to which each 
of the comprehensive centers meets the objectives of its respective plan, and whether such 
services meet the educational needs of State educational agencies, local educational agencies, 
and schools in the region.” The program evaluation was conducted by Branch Associates, Inc., 
Decision Information Resources, Inc., and Policy Studies Associates, Inc. 

With the redesign of the Center program, the primary focus of technical assistance was 
directed to states. In order to build states’ capacity for carrying out NCLB responsibilities, which 
include assistance to struggling school districts and schools as well as other areas of NCLB 
program administration, the Center program was designed to supply ongoing technical assistance 
in using research knowledge and promising practices. There are two types of Centers: 

■ Sixteen Regional Comprehensive Centers (RCCs) are responsible for providing 
ongoing technical assistance to states assigned to their region, working with a 
range of one to eight states per Center 

■ Five Content Centers (CCs) are expected to supply knowledge to RCCs and work 
with RCCs to assist states in the CC’s specialty area: Assessment and 
Accountability, Instruction, Teacher Quality, Innovation and Improvement, or 
High Schools 

Given this program design, the evaluation provides a description of Center operations. It also 
reports on assistance delivery and contributions to state capacity as judged by managers in state 
education agencies (SEAs), on quality as judged by panels of subject-matter experts, and on 
relevance and usefulness as judged by practitioners who participated in Center activities or 
received Center products. The evaluation data, collected annually, pertain to the Center program 
years 2006-07, 2007-08, and 2008-09, covering three of the five program years starting with the 
second year of program funding. 1 

■ The operations of the RCCs and CCs were consistent with the Center program 
design. RCCs and CCs assessed client needs annually to determine their technical 
assistance plans, with informal communications as the mode most commonly 
reported for 2008-09. The most common activity found in sampled RCC projects 2 


1 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106), 
32583-94. The awards were subsequently extended. 

2 For the purposes of this evaluation, the team identified “projects” as a common level of aggregation of Center 
activities that would constitute units large enough for review and rating, but focused enough for coherence. A 
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was “ongoing consultation and follow up” (82, 93, and 91 percent of the sampled 
RCC projects in years 2006-07, 2007-08, and 2008-09, respectively), consistent 
with the charge to provide frontline assistance on an ongoing basis to states. In 
CC projects the most common activity was “research collections and synthesis” 
(more than 70 percent of sampled projects in each year), consistent with the CCs’ 
prescribed focus on synthesizing, translating, and delivering knowledge to RCCs 
and states. Across the three years studied, both RCCs and CCs were more 
involved in each other’s projects. Among sampled RCC projects, the percentage 
that included direct assistance from CC staff was 18 percent in 2006-07, 22 
percent in 2007-08 and 30 percent in 2008-09. The percentage of CC projects that 
included RCC direct assistance was 1 1 percent in 2006-07, 12 percent in 2007-08, 
and 38 percent in 2008-09. In addition, by 2008-09 all 16 RCCs reported 
receiving knowledge resources from CCs and all 5 CCs reported providing 
knowledge resources to RCCs. 

■ Centers addressed the most frequently cited state priority of “ statewide systems 
of support,” and an increasing number of state managers reported each year 
that Center assistance served their purposes. “Systems of support” consists of an 
infrastructure for the delivery of onsite assistance, and strategies and materials 
designed to help struggling schools and districts improve student performance. 
The most widespread NCLB -related priority for state managers was “statewide 
systems of support or school support teams,” which was identified as a major or 
moderate priority for technical assistance by more than 90 percent of managers, 
weighted, in each year. Of this group of state managers, more than 90 percent 
reported each year that the Centers delivered assistance related to this 
responsibility. “Systems of support” was not only the most widely reported state 
priority but also the topic addressed in more Center projects in each year than any 
other topic, according to the inventories compiled by the Centers (19 percent of 
all projects in 2006-07, 25 percent in 2007-08, and 21 percent in 2008-09, 
compared with 10 percent or fewer projects addressing any other topic). With 
each state weighted equally in the analysis, the proportion of state agency 
managers reporting that assistance from the Centers had “served the state’s 
purposes completely” rose from about one-third (36 percent) in 2006-07 to more 
than half (56 percent) in 2008-09. 

■ Center assistance was reported by state managers as having expanded state 
capacity in “ statewide systems of support,” which has been a predominant focus 
of Center assistance. Among state managers who reported statewide systems of 
support or school support teams as a state priority for technical assistance in 
2008-09, 82 percent credited Center assistance with a “great” or “moderate” 
expansion of state capacity in this area. In other areas of state responsibility 
identified by state managers to be a priority for technical assistance, the 
percentage reporting a great or moderate expansion of state capacity in 2008-09 
ranged from 77 percent (for research-based curriculum, instruction, or 


“project” was defined as a group of closely related activities and/or deliverables designed to achieve a specific 
outcome for a specific audience. 



professional development in academic subjects) to 39 percent (for NCLB’s 
provisions on supplemental educational services and choice). 

■ On average across each of the three years, expert panels rated sampled project 
materials between “ moderate ” and “high” for quality, and project participants 
rated the sampled projects “high” for relevance and usefulness. Pro gram- wide 
average ratings, on a 5 -point scale with 5 at the high end, were 3.34 in 2006-07, 
3.51 in 2007-08, and 3.57 in 2008-09 for technical quality; 3.94, 4.08, and 4.15, 
respectively, for relevance; and 3.69, 3.95, 3.96, respectively, for usefulness. 3 In 
addition, the average quality rating was consistently higher among CC projects 
than RCC projects by more than one-half of a standard deviation while RCC 
ratings went up each year. 4 The average ratings of relevance were higher for RCC 
than CC projects in 2006-07 and 2007-08 although CC ratings went up each year; 
there were no consistent differences in the usefulness ratings between RCCs and 
CCs. 


The Comprehensive Centers Program 

In its authorization under the Educational Technical Assistance Act of 2002, the Center 
program was given an overall charge of supporting state and local NCLB implementation. The 
U.S. Department of Education (ED), using discretion provided in the legislation, established two 
major program features that differed from the design of Comprehensive Center programs under 
prior legislation: 5 

■ First, the primary focus would be on assisting states to carry out NCLB 
responsibilities and helping build state capacity to deliver assistance to schools 
and districts; ED specified that Centers could only work directly with districts or 
schools under special circumstances. 

■ Second, awards would be made to 21 Centers to establish two-tiers of technical 
assistance with 16 RCCs and 5 CCs. They were instructed to work as follows: 

■ Each RCC was charged with providing ongoing assistance directly to 

states in its region (“frontline assistance”), serving the needs of either one 


3 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 
to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 
and each project sampled from a Center contributed equally to the Center mean. 

4 All project-level differences described in this report (e.g., more, higher) reflect a difference of one-half of one 
pooled standard deviation between groups of projects. Using a metric derived from Cohen ( 1988), the evaluation 
team estimated Cohen's d (an estimate of the effect size defined as the difference in means divided by the pooled 
standard deviation) and adopted the logic of Cohen for what would be considered a moderate difference. For this 
study, inferential tests of statistical significance were not conducted to examine project-level differences in these 
non-probability samples. All participant-level differences described in this report reflect statistical test of 
significance with a criterion value of p<.05. 

5 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106), 
32583-94. 
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large state or a group of two to eight states and other jurisdictions. 6 The 
RCCs were also expected to deliver technical assistance to their assigned 
states, addressing the needs and building capacity of the states to assist 
their districts and schools. 

Meanwhile, each CC would work on a nationwide basis to provide in- 
depth knowledge of the content and research within a particular 
substantive area: Assessment and Accountability, Instruction, Teacher 
Quality, Innovation and Improvement, or High Schools. CCs would 
facilitate access to, and use of, existing research and practices. 

The absolute priorities for the two types of Centers indicated that they 
should work together: Regional Centers should draw information and 
resources from Content Centers as well as other sources; and Content 
Centers should both supply knowledge to Regional Centers and “work 
closely with Regional Centers to provide technical assistance to States.” 


Research Questions and Methods 

The research priorities for the evaluation were primarily driven by the statute and focused 
on the following key research questions: 

1. How did the Regional Comprehensive Centers and Content Centers operate as part of the 
Comprehensive Technical Assistance Center program? 

■ How did Centers develop, refine, and carry out their plans for technical 
assistance? How did they define their clients’ educational needs and priorities? 

■ What were the objectives of the technical assistance the Centers offered? What 
kinds of products and services were provided by the Centers? 

■ How did the Regional Comprehensive Centers and Content Centers coordinate 
their work? 

2. What was the performance of the Comprehensive Centers in addressing state needs and 
priorities? How did their performance change over the period of time studied? 

■ How did the Centers’ state clients define their needs and priorities? 

■ To what extent, as reported by states, did Center assistance expand state capacity 
to address underlying needs and priorities and meet the goals of NCLB? 

6 The nonstate jurisdictions that the Centers were to serve were the following: the District of Columbia, Puerto Rico, 
the Virgin Islands, American Samoa, Commonwealth of the Northern Mariana Islands, Federated States of 
Micronesia (Chuuk, Kosrae, Pohnpei, and Yap), Guam, Republic of the Marshall Islands, and Republic of Palau. 
Throughout this report, the term “state” will be defined to include the 50 states as well as these other jurisdictions. 
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■ To what extent did states rely on other sources of technical assistance besides the 
Centers? What were other sources of technical assistance that states used? How 
did the usefulness of Center assistance compare with the usefulness of assistance 
from other sources? 

3. To what extent was the assistance provided by the Centers of high quality, high relevance, 
and high usefulness? 

■ Did the quality, relevance, or usefulness of Center assistance change over the 
period of time studied? 

■ What was the variation in the quality, relevance, and usefulness of Center 
assistance across types of projects and participants? 


The evaluation gathered information annually on the Center program for the years 2006- 
07, 2007-08, and 2008-09 from six data sources in order to address the research questions above. 
Data sources included: 

■ Management plans. The evaluation reviewed these plans as a data source for 
each Center’s intended focus at the beginning of the year, drawing from the plans 
a list of topics as foci of Center objectives. 

■ Project inventory forms and cover sheets. Each Center completed an inventory 
of its work that grouped related activities and deliverables into “projects,” with 
the project defined as a group of closely related activities and/or deliverables 
designed to achieve a specific outcome for a specific audience. Projects were in 
turn classified by the Centers into major, moderate, and minor projects on the 
basis of the relative level of effort they reflected. The Centers also classified the 
projects, according to the topics addressed, into 22 topical categories. 7 At each 
stage, the evaluation team provided written guidance and training for inventory 
development, reviewed the Centers’ drafts, and clarified definitions as needed. 
For projects sampled for the evaluation, the Centers prepared “cover sheets” 
providing brief descriptions and contexts for the activities and resources included 
in the project. The evaluation team used the cover sheets as a data source for 
coding project activities and resources. 


7 The 22 topics were: components of effective systems of support for states, districts, and schools; data use or data- 
driven decision making; formative assessment; reading; adolescent literacy; mathematics; dropout prevention; high 
school redesign or reform; transition to high school; special education curriculum, instruction and professional 
development; special education assessment; English language learners;” highly qualified teacher” provisions of 
NCLB; teacher preparation and induction; teacher professional development; supplemental educational services; 
Response to Intervention; migrant education; Indian or Native American education; data management and 
compliance; assessment design; and parent involvement. In addition, projects that addressed none of these 22 topics 
were categorized as “other.” 
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■ Center staff interviews. Using structured response categories, Center staff were 
asked about how they planned their programs of work; how their plans evolved 
during the program year; and what they offered to clients with respect to the 
topics addressed, the delivery modes used, and their sources for content expertise. 

■ Survey of senior state managers. SEA managers were surveyed about their 
state’s technical-assistance needs and what was provided by the Centers 
(including their RCC and any CCs with whom their state had worked). 

■ Expert panel review. The same sample of major and moderate projects was 
reviewed for quality by a panel of experts. Content experts were recruited and 
trained to use standard criteria to rate the technical quality of the sampled Center 
projects on the basis of a review of all project materials. 

■ Survey of project participants. A representative sample of clients who had 
participated directly in the evaluation’s purposive sample of major and moderate 
Center projects furnished descriptive information, through surveys, on the 
activities and resources that the project had delivered to them. These clients 
included individuals working at the state level who had participated in RCC or 
CC projects, and RCC employees who were among the clients of CC projects. 
They rated the relevance and usefulness of the sampled projects. 


Center Operations 

In designing the Center program, ED established structures and expectations for the 
functioning of the Centers. Key features of the design, emphasized in ED’s Notice Inviting 
Applications, were a requirement for needs assessment in consultation with clients, a focus on 
technical assistance with state responsibilities in school and district improvement, and the 
specialized roles of RCCs and CCs. The Centers’ work from 2006-07 to 2008-09 conformed to 
the program’s requirements in each of these respects. The barriers to technical assistance that 
Centers most often reported were staff turnover in state agencies and limitations on the CCs’ 
scope of work. 

A key expectation of the Centers was to organize their plans around the priorities and 
needs of client organizations. At the start of each program year, the Centers were required to 
deliver a management plan to ED outlining the program of technical assistance they planned to 
provide. Across years, Centers used a range of methods to assess needs and plan technical 
assistance with their clients. Among RCCs, there was a shift away from conducting surveys for 
needs assessment while maintaining frequent interaction with states as a means of learning about 
needs: all 16 RCCs reported assessing needs for 2008-09 through ongoing communication with 
state staff (an increase from 15 RCCs in 2006-07); 15 had a designated liaison to the SEA on 
staff in 2008-09 (up from 13 in 2006-07); 8 of the 16 conducted surveys (down from 11). Across 
years, all 5 of the CCs reported forming their work plans with RCC input acquired through 
ongoing communication; in 2008-09 all 5 CCs reported surveying RCCs (up from 4 of the 5 in 
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2006-07). In addition, all 5 CCs reported learning about state needs for 2008-09 through ongoing 
interaction with states as well as through communication with RCCs. 

Centers were expected to show responsiveness to needs and requests for technical 
assistance but might not be in a position to respond to every client request. In each year, more 
than half of the Centers reported that they had turned down a client request for assistance, a 
situation that was handled differently by RCCs and by CCs. The number of Centers that reported 
having turned down one or more requests was 12 of the 21 in 2006-07, 13 of 21 in 2007-08, and 
14 of 21 in 2008-09. Among Centers that declined any client request for assistance in 2008-09, 
RCCs most often reported substituting a different type of assistance (7 of the 10 RCCs that 
turned down work used this strategy), but none of the CCs reported doing so. The reason most 
commonly reported by RCCs was that a request fell outside their legitimate scope of work (5 
RCCs vs. 1 CC). CCs reported more concerns with the requests fitting the Center’s priorities (2 
of the 4 CCs that turned down work vs. 2 of the 10 RCCs) or the Center’s capacity (2 of the 4 
CCs that turned down work vs. 2 of the 10 RCCs). 

The Centers placed a priority on assistance with the state role in supporting improvement 
in struggling schools and districts. In every year of the evaluation, on the inventories completed 
by Centers that grouped their technical assistance activities into projects and categorized projects 
into 23 topics, the most common topic for all Center projects was “components of effective 
systems of support — state, district, school,” a topic that included but was not limited to statewide 
systems of support and school support teams. Among all projects on the Center’s inventories, 19 
percent in 2006-07, 25 percent in 2007-08, and 21 percent in 2008-09 addressed the topic of 
systems of support, which in each year was more than twice as many as any other topic. 

Although the two types of Centers each retained a focus on activities distinctly associated 
with the original program design, their ways of working became more similar over the years. The 
guidance given by ED through the Center grant competition and afterwards laid out a particular 
structure for the Centers’ work: RCCs would specialize in interactions with state clients while 
CCs would specialize in activities that required a content focus. The most common activity 
found in sampled RCC projects was “ongoing consultation and follow up” (82 percent in 2006- 
07; 93 percent in 2007-08; 91 percent in 2008-09); in CC projects, it was “research collections 
and synthesis” (74 percent in 2006-07, 85 percent in 2007-08, and 77 percent in 2008-09), while 
fewer RCC projects included this activity (53 or 54 percent in each year) (exhibit ES.l). In 2008- 
09, in a departure from past CC practice, a majority of sampled CC projects (62 percent) 
included ongoing consultation and follow-up. 
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Exhibit ES.1. Sampled RCC and CC projects by types of activities and 
resources, by year 



Percent of sampled RCC 
projects 

Percent of sampled CC 
projects 

Activities and resources 

2006-07 

(n=96) 

2007-08 

(n=96) 

2008-09 

(n=93) 

2006-07 

(n=27) 

2007-08 

(n=26) 

2008-09 

(n=26) 

Ongoing consultation and follow- 
up 

82% 

93% 

91% 

22% 

15% 

62% 

Research collections and 
syntheses 

54 

53 

53 

74 

85 

77 

Engagement of participants in 
project planning 

45 

63 

59 

30 

31 

27 

Training events 

43 

55 

59 

37 

50 

50 

Task force meetings and work 

50 

58 

56 

7 

8 

8 

Conferences 

27 

35 

40 

63 

42 

38 

Support development of a formal 
plan to implement a program or 
policy 

19 

32 

31 

7 

15 

23 


EXHIBIT READS: For the 2006-07 program year, 82 percent of sampled Center projects included ongoing 
consultation and follow-up. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. In addition to serving 
as resource material for the expert reviewers, these cover sheets were coded by the evaluation team. 


The delivery of technical assistance depended on the Centers working effectively with 
their clients. Both RCCs and CCs described the barriers they perceived as having impeded their 
assistance to states. Turnover in staff within state offices or intermediary units was reported by 
both types of Centers as a barrier to achieving their objectives in assisting states (10 of 16 RCCs 
and 3 of 5 CCs). Turnover at the leadership level was a reported barrier for 8 RCCs and 3 CCs. 
Three of the 5 CCs reported “a state’s most important priorities for assistance fell outside the 
Center’s scope of work,” as a barrier; they indicated that some states wanted help with topics that 
went beyond their own assigned substantive focus. 

Under the two-tiered Center program design, RCCs and CCs were expected to work 
together to serve state clients. Among RCC projects, 48 percent had a CC contribution (of 
materials, in-person assistance, or advice) in 2006-07, 32 percent in 2007-08, and 47 percent in 
2008-09. Among CC projects, the percent incorporating some RCC contribution was 37 percent, 
38 percent, and 42 percent across the years. The extent to which RCCs and CCs drew on the 
other as substantive partners in delivering assistance increased in 2008-09: the percent of 
sampled RCC projects in which CCs delivered technical assistance went up from 18 percent in 
2006-07 to 30 percent in 2008-09, and the percent of sampled CC projects in which the RCCs 
delivered technical assistance rose from 1 1 percent in 2006-07 to 38 percent in 2008-09. 
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With 16 RCCs and 5 CCs all charged with working with the other type of Center, 
coordination varied across the different pairs of an RCC and a CC. For example, while 15 of the 
16 RCCs reported teaming up with at least one CC to provide technical assistance to states, 14 of 
them reported teaming up with one of the CCs but 7 of them reported doing so with another of 
the CCs. In addition, CCs were expected to provide assistance to RCCs, and the barrier most 
often reported by both types of centers to have impeded CCs’ achievement of their technical 
assistance objectives with RCCs was that “RCCs’ most important priorities for assistance fell 
outside the CC’s scope of work” (reported as a barrier by 7 of 16 RCCs and 4 of 5 CCs). 


Extent to Which Centers Addressed State Priorities 

The perceptions of senior managers in state education agencies, who are involved in 
identifying state needs and priorities for technical assistance, provide a relevant perspective on 
the outputs of the program. Because the Centers had a mandated focus on the states, the extent to 
which state managers perceived that Center technical assistance served state purposes is one way 
of gauging the program’s attainment of its objectives. Capacity building for states is also a focus 
of this evaluation, because it was prominent as a goal for the Comprehensive Centers program. 
The first priority for all Centers, articulated by ED in the Notice Inviting Applications, included 
“helping states build the capacity to help school districts and schools implement NCLB 

o 

provisions and programs.” 

An increasing percentage of state managers (weighted) over three years reported that the 
Centers’ technical assistance “served the state’s purposes completely” (exhibit ES.2). Thirty-six 
percent of the state managers, weighted, chose this response for 2006-07, 47 percent for 2007-08, 
and 56 percent for 2008-09. 

Among the managers who said their state’s purposes were not completely served, a larger 
proportion in each year reported that they wanted more interaction with the Centers. The percent 
of weighted state managers saying, “Center staff are not able to spend as much time working 
with the state as we would like” was 17 percent of those who did not say the state’s purposes 
were “completely” served in 2006-07. The corresponding figures for subsequent years were 27 
percent in 2007-08 and 43 percent in 2008-09. 8 9 (These respondents were 9 percent, 10 percent, 
and 16 percent, respectively, of all state managers, weighted.) 


8 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, tune 3). 70(106), 
32585. 

9 The percentage of state managers reporting that their state’s purposes were not completely served varied by year. 
Thus, for the follow-up question (“reasons the technical assistance has been less helpful than it might be”), 
comparison of percentages from year to year may include variation in responses over time as well as changes in 
respondents addressing the question. 
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Exhibit ES.2. Extent to which technical assistance from the Centers served 
state purposes, as judged by state managers, by year 



It did not serve the state's 
purposes 

It was a good start, but some 
important priorities were 
not addressed 


It was a good start 


It served the state's 
purposes completely 


2006-07 2007-08 2008-09 

(n=56) (n=57) (n=54) 


EXHIBIT READS: In 2006-07, 36 percent of state managers, weighted, reported that Center 
technical assistance served the state’s purposes completely. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally 
represented in instances where more than one manager from a state responded. 


State managers in a high proportion of states reported that Centers delivered assistance on 
“statewide systems of support or school support teams,” which was the most widespread priority 
among areas of technical assistance for state managers. In 2007-08 and 2008-09, more than 
90 percent of state managers, weighted, identified this area of state responsibility as a major or 
moderate priority for technical assistance (95 percent in 2007-08, 94 percent in 2008-09). Of this 
group of state managers that reported this priority 10 , more than 90 percent reported that the 
Centers delivered assistance related to this responsibility (94 percent in 2007-08, 91 percent in 
2008-09). 

Looking at state reported capacity building across areas of major or moderate state 
priority for technical assistance, the highest percentage of state managers reported Center 
assistance to have expanded their agency’s capacity to a “great” or “moderate” extent in 
statewide systems of support or school support teams 11 (72 percent in of those who rated the area 


10 The subgroup of state managers who identified a particular area of state responsibility to be a major or moderate 
priority for technical assistance varied by year. Thus, for the follow-up question about the priority areas in which 
states received technical assistance from Centers, a comparison of the percentages from year to year may include 
variation in responses over time as well as changes in respondents addressing the question. 

1 1 Percentages are based on the state manager respondents who rated each area as major or moderate technical 
assistance priority, which varied by year. Thus, for the question about state capacity building, comparison of 
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as a state priority in 2007-08 and 82 percent in 2008-09) (exhibit ES.3). The next-highest in both 
2007-08 and 2008-09 was “development or dissemination of research-based curriculum, 
instruction, or professional development programs in academic subject(s)” (64 percent in 2007- 
OS and 77 percent in 2008-09). In both years, the lowest was “administration of supplemental 
educational services and choice provisions” (44 percent and 39 percent, respectively, of those 
who rated the area as a priority), which was an area rated as a priority by the fewest state 
managers (49 percent and 48 percent, respectively). 


Exhibit ES.3. Extent to which Center assistance expanded state capacity in 

priority areas, as judged by state managers who rated the area as 
a major or moderate technical assistance priority, by year 



Percent reporting 
capacity expanded by a 
great or moderate extent 

Area of state responsibility under NCLB 

2007-08 

2008-09 

Statewide systems of support or school support teams 
(n=56, n=50) 

72% 

82% 

Policies and practices for English language learners 
(n=43, n=40) 

59 

73 

State assessment and accountability systems 
(n=42, n=39) 

57 

59 

Development or dissemination of research-based 
curriculum, instruction, or professional development 
programs in academic subject(s) 

64 

77 

(n=41 , n=39) 

Assistance with educators’ use of assessment data 
(n=37, n=36) 

62 

61 

Monitoring compliance with NCLB requirements 
(n=35, n=30) 

57 

57 

Administering supplemental educational services (SES) 
and choice provisions (n=25, n=26) 

Communication with parents or the public 
(n=25, n=26) 

44 

39 

48 

50 


NOTE: Percentages are based on the state manager respondents who rated each area a major or moderate 
technical assistance priority, which varied by year. State managers who chose the response, “Does not 
apply, or not able to judge,” were included in the denominator of the percent calculation. 

EXHIBIT READS: For the 2007-08 program year, among the 56 state managers (weighted) who reported 
that technical assistance in statewide systems of support or school support teams was a major or moderate 
priority for their state, 72 percent reported that technical assistance received from the Centers expanded the 
state’s capacity in this area to a great or moderate extent. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in 
instances where more than one manager from a state responded. 


percentages from year to year may include variation in responses over time as well as changes in respondents 
addressing the question. 
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State managers reported that the Centers were one of multiple sources that they used for 
technical assistance, but over time they turned to the Centers for more purposes. The purposes 
for which states used the Centers more than other sources in each of the three study years were 
“to plan the initial steps in solving a problem” (reported as a purpose for Center technical 
assistance by at least 60 percent of state managers in each year) and “to develop the skills of 
SEA or intermediate education agency staff’ (at least 55 percent of state managers in each year). 
In 2007-08 and 2008-09, Centers were reported by state managers to be the top source for two 
additional purposes: to help states complete tasks where they lacked resources (58 percent and 64 
percent, respectively) or expertise (49 percent and 53 percent, respectively). 


Ratings of Center Assistance 

To assess the technical assistance provided by the Center program, quality, relevance, 
and usefulness of a sample of Center projects were rated. All sampled projects were identified by 
the Centers as “major” or “moderate” in their level of effort, relative to other projects in the same 
Center. The projects were rated for technical quality by panels of experts with strong knowledge 
of the content or substantive focus of the specific projects they reviewed. Projects’ relevance and 
usefulness were rated by a sample of participants — state staff, intermediate agency staff, local 
educators working on behalf of the state, and RCC staff — who were the intended beneficiaries of 
the project and who had received at least some of the technical assistance the project provided. 
Quality was judged on three dimensions; relevance was assessed with eight survey items and 
usefulness with 11 survey items (exhibit ESA). Each overall measure (relevance, usefulness, or 

quality) was calculated as the mean of ratings assigned to each item. The item-level ratings 

12 

themselves were based on 5-point rating scales. 

Based on the ratings, Center technical assistance was rated higher on each measure in 
each successive year, with program- wide average ratings in 2008-09 falling in a range between 
“moderate” and “high” for quality, and around “high” for relevance and usefulness (exhibit 
ESA). On a scale of 1 to 5 with a 3 representing “moderate” and a 4 representing “high,” the 
program-wide average ratings for the sampled projects were 3.34 in 2006-07, 3.51 in 2007-08, 
and 3.57 in 2008-09 for technical quality, scored by panels of content experts. Program-wide 
average ratings for relevance, scored by participants, were 3.94 in 2006-07, 4.08 in 2007-08, and 
4.15 in 2008-09. Average usefulness ratings for the program were 3.69 in 2006-07, 3.95 in 2007- 
OS, and 3.96 in 2008-09, also scored by participants. 13 


12 Efforts were made to develop parallel wording and rubrics that would result in similar gradations between rating 
levels (e.g., very high vs. high vs. moderate) across the three measures. However, given the different content of each 
set of items within the three measures and the different contexts for the ratings (experts who underwent training for 
the rating process and reviewed identical packages of materials vs. survey respondents who typically participated in 
different subsets of project activities), the ratings across the three measures are not directly comparable. 

13 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 
to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 
and each project sampled from a Center contributed equally to the Center mean. 
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Exhibit ES.4. 


Quality, relevance, and usefulness items 


From expert panel scoring From project participant surveys 

Technical quality Relevance Usefulness 


Reviewers were directed to 
assign a score to each 
dimension and to include the 
basis for their ratings on the 
rating form, including the specific 
artifacts on which their score was 
based. The three dimensions 
are: 

a. Demonstrated use of the 
appropriate documented 
knowledge base - to include 
an accurate portrayal of the 
current state of information 
with prominence to those 
with the most 

accurate/rigorous evidence 

b. Fidelity of application of the 
knowledge base to the 
products and services 
provided - materials are 
consistent with the 
best/accurate information 
available and the 
presentation adequately 
conveys the confidence of 
the information 

c. Clear and effective delivery - 
information is well organized 
and written and accessible to 
the intended audience for 
easy use 


Based on your experience, to 
what degree was this set of 
activities and resources relevant 
to your work, in each of the 
following respects? 

a. Addressed a need or problem 
that my organization faces 

b. Addressed an important 
priority of my organization 

c. Addressed a challenge that 
my organization faces related 
to the implementation of NCLB 

d. Provided information, advice, 
and/or resources that could be 
directly applied to my 
organization’s work 

e. Addressed our particular state 
context 

f. Addressed my organization’s 
specific challenges (e.g., 
policy environment, leadership 
capacity, budget pressures, 
local politics) 

g. Provided information, advice, 
and/or resources that could be 
used to guide decisions about 
policies, programs, or 
practices 

h. Highlighted the implications of 
research findings (or 
information about best 
practice) for policies, 
programs, or practices 


Based on your experience, to 
what degree was this set of 
activities and resources useful to 
you, in each of the following 
respects? 

a. Provided resources that were 
easy to understand and easy 
to use 

b. Employed an appropriate 
format (e.g., a work group, a 
conference, individual 
consultation, written products) 

c. Provided adequate opportunity 
to learn from colleagues in 
other states 

d. Included adequate follow-up to 
support the use of new 
information and resources 

e. Were timely 

f. Helped my organization solve 
a problem 

g. Helped my organization 
maintain or change a policy or 
practice 

h. Helped my organization take 
the next step in a longer-term 
improvement effort 

i. Provided my organization with 
information or resources that 
we will use again 

j. Helped my organization 
develop a shared expertise or 
knowledge-base 

k. Helped individuals in my 
organization to develop skills 
that they will use again 
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Exhibit ES.5. Mean ratings of technical quality, relevance, and usefulness, 
by center type and by year 



Technical quality 

Relevance 

Usefulness 


2006- 

2007- 

2008- 

2006- 

2007 - 

2008- 

2006- 

2007 - 

2008- 


07 

OS 

09 

07 

08 

09 

07 

08 

09 

All Centers (N=21) 

3.34 

3.51 

3.57 

3.94 

4.08 

4.15 

3.69 

3.95 

3.96 

All RCCs (N=16) 

3.21 

3.41 

3.52 

3.99 

4.18 

4.15 

3.71 

3.99 

3.94 

All CCs (N=5) 

3.73 

3.86 

3.72 

3.78 

3.96 

4.17 

3.65 

3.84 

4.01 

Difference of RCC 
and CC means 

-0.52 t 

-0.45 t 

-0.20 t 

0.21* 

0.22 t 

-0.02 

0.06 

0.15* 

-0.07 

Pooled standard 
deviation (all Centers) 

0.41 

0.41 

0.37 

0.34 

0.22 

0.24 

0.34 

0.23 

0.27 

Ratio of difference in 
means to pooled 
standard deviation 

- 1.28 

- 1.09 

- 0.55 

0.62 

1.00 

- 0.08 

0.18 

0.64 

- 0.26 


NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” rating is the mean of the 
ratings for the three quality dimensions. A notation of f indicates that the difference in the mean ratings between the 
CCs and RCCs within that year is at least one-half of one pooled standard deviation in the rating. 


EXHIBIT READS: In 2006-07 among the 21 Centers, the mean technical quality rating was 3.34. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to project 
ratings; each project contributed equally to Center ratings; and each Center contributed equally to cross-Center 
ratings. 


Given that the RCC and CC roles and activity emphases differed, the evaluation looked at 
variation across Center types. The mean ratings for types of Centers, based on their sampled 
projects, showed the CCs with higher mean ratings than RCCs for the quality of their sampled 
projects in all three years although RCCs’ average quality ratings were higher in each successive 
year (exhibit ES.5). The RCCs had higher mean ratings than CCs for the relevance of their 
sampled projects in 2006-07 and 2007-08 although the average ratings of relevance for CCs went 
up each year. There were no consistent differences in mean ratings of usefulness across types of 
Centers. 

The evaluation also looked at the relationships between the three measures: quality, 
relevance, and usefulness. It was reasoned that the content experts rating quality and the 
participants rating relevance and usefulness might be better able to judge different aspects of a 
Center project. On this rationale, content experts rated the projects for their technical quality, and 
participants rated the projects for relevance and usefulness. An examination of the associations 
among the three dimensions was conducted by calculating correlation coefficients. 14 Such a 
statistic indicates the strength and direction of a linear relationship between two factors. A 
correlation coefficient can vary from positive 1.00 (indicating a perfect positive relationship), 


14 For this analysis, the evaluation team used Spearman’s rank order correlation, as this non-parametric rating is the 
appropriate statistical function to describe correlations between two variables where the values of the variables are 
not normally distributed and are on a scale (such as ratings). 
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through zero (indicating the absence of a relationship), to negative 1.00 (indicating a perfect 
negative relationship). If the correlation is statistically significant (p <.05), we can have strong 
(95 percent) confidence that what we calculated is not due to chance. 

In every year, ratings of quality were unrelated to ratings of relevance and usefulness, 
although relevance and usefulness ratings were highly correlated with each other within each of 
the three data collection years. The correlation coefficient for relevance and usefulness was 
+0.84 for 2006-07, +.79 for 2007-08, and +.83 for 2008-09. This indicates that the extent to 
which participants rated the projects as relevant was associated with how they deemed the 
project to be useful to their agency. These coefficients were all statistically significant at p<.05. 
On the other hand, the results indicated correlations ranging from -0.12 to +0.04 between quality 
and relevance, and from -0.09 to +0.07 between quality and usefulness. Because these 
coefficients are not statistically significant we cannot be sure that they are different from zero 
(no relationship). In other words, the extent to which a project faithfully reflected the knowledge 
base on a topic and provided appropriate caveats about the quality of its evidence was unrelated 
to the extent to which participants deemed that project relevant or useful to their agency. 

Given the variation in ratings across Centers, additional analyses were conducted to 
explore whether there were consistent patterns between ratings and the particular features of the 
projects. Such information could provide suggestions for possible program improvement if there 
were consistent relationships. Quality ratings in 2008-09 were higher for RCC projects that 
included CC contributions of materials or in-person help than projects that the RCCs completed 
without CC contributions (3.72 vs. 3.39), although this was not the case in earlier years. In 
addition, quality ratings were higher in 2008-09 for projects that had been reviewed by CCs 
(3.83 vs. 3.46) and by outside experts (3.73 vs. 3.42) for quality assurance as opposed to projects 
that had not been reviewed in each of these ways (a project-level feature that was studied only in 
that year of the evaluation). In other analyses of project-level variation, projects that differed 
from each other in the activities they encompassed or the topics they addressed did not show 
differences in ratings of quality, relevance, or usefulness that were consistent across the three 
years. 


On the other hand, more consistent differences were found in ratings of relevance and 
usefulness awarded to projects by different types of participants. Higher ratings were awarded by 
those participants who had been involved in determining the project goals or design than by 
participants not involved in this way, and by those who had spent more time in project activities 
(i.e., 6 or more days) as compared to participants who had spent five days or less (these 
differences were statistically significant, with p <.01 for both relevance and usefulness). For 
2007-08 and 2008-09, also, each type of Center targeted its assistance more successfully to 
participants who worked in one type of agency, compared with participants who worked in other 
types of agencies: specifically, RCC projects were rated higher by participants from SEAs than 
participants from intermediate or local education agencies or schools; CC projects were rated 
higher by RCC staff than by SEA staff (statistically significant differences, with p<.05 for both 
relevance and usefulness). 
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1. Introduction and Background 


The Comprehensive Technical Assistance Centers are authorized under the Educational 
Technical Assistance Act of 2002 to provide technical assistance for implementation of the No 
Child Left Behind (NCLB) Act at the state, district, substate region, and school levels for the 
purposes of “improving academic achievement, closing achievement gaps, and encouraging and 
sustaining school improvement” (Section 203). The law authorized the Secretary of Education to 
award “not less than 20 grants to local entities, or consortia of such entities, with demonstrated 
expertise in providing technical assistance and professional development in reading, 
mathematics, science, and technology” (Section 203). Grants were awarded to 21 Centers to 
serve different geographic regions across the United States from FY2006 through FY2010. 

The same law authorizing these technical assistance centers also mandated that the 
Comprehensive Technical Assistance Centers undergo independent evaluation under the 
direction of the National Center for Education Evaluation and Regional Assistance (NCEE) in 
the Institute of Education Sciences of the U.S. Department of Education (ED). The legislation 
indicated that the evaluation should “include an analysis of the services provided... [and] the 
extent to which each of the comprehensive centers meets the objectives of its respective plan, 
and whether such services meet the educational needs of State educational agencies, local 
educational agencies, and schools in the region.” 

This is the final report of a multi-year evaluation conducted for NCEE by Branch 
Associates, Inc., Decision Information Resources, Inc., and Policy Studies Associates, Inc. This 
final report presents an overall description of how the system of technical assistance centers has 
operated from July 2006 to June 2009, covering the period from the second to the fourth program 
year (out of the five years), as part of the evaluation to inform the current grants as well as the 
grant recompetition. 

This introductory chapter provides background information on ED’s purposes and design 
for the Center system. It also describes key state responsibilities included in the No Child Left 
Behind (NCLB) Act with which the Centers were expected to provide assistance. The chapter 
concludes with a description of the major evaluation questions. 


The Center Program 

The current Center program represented a departure from the previous program with a 
new design for targeting services replacing the 15 existing Comprehensive Regional Assistance 
Centers established under the Elementary and Secondary Education Act of 1994 (ESEA). The 
1994 reauthorization of ESEA had charged the previous set of 15 Centers with delivering 
assistance to support standards-based reform as envisioned in other sections of the 1994 law. 
They were to provide training and technical assistance to states, local education agencies 
(LEAs), schools, tribes, community-based organizations, and other ESEA grantees related to 
several areas of local responsibility. These included: (1) improving the quality of instruction, 
curricula, assessments, and other aspects of school reform; (2) implementing effective 
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schoolwide programs; and (3) meeting the needs of children, especially children in high-poverty 
areas, migrant children, immigrant children, limited-English-proficient children (LEP), neglected 
or delinquent children, homeless children, Indian children, and children with disabilities [Section 
13102 (a)(l)(A-L)]. In short, the previous Centers’ mandate included a focus on a number of 
aspects of local educational practice. 

A previous evaluation 15 found the majority of the technical assistance was targeted to 
school districts and schools. The study, based on surveys of clients conducted in 1999, found that 
the majority of direct participants in major Center training and technical assistance activities 
were school staff (either teachers or principals), and that the majority of clients who arranged for 
services from the Centers were also based in school districts or schools. At the state level, among 
those state education agency (SEA) staff members who had received assistance, 64 percent 
reported that the Centers had improved the ability of their SEA to provide assistance to districts 
and schools. The report noted that higher ratings for Centers were associated with more intensive 
technical assistance, based upon survey findings. 

The current design of the Center program reflects changes from the design of the 
predecessor program. The Educational Technical Assistance Act of 2002 authorized a new group 
of Centers to provide technical assistance for NCLB implementation at the state, district, substate 
region, and school levels for the purposes of “improving academic achievement, closing 
achievement gaps, and encouraging and sustaining school improvement” (Section 203). It also 
gave ED discretion to “establish priorities” for the Centers (Section 207). 

In making its design choices, ED set up an advisory process to identify priorities for the 
new Centers: in 2004 the Secretary of Education appointed 10 Regional Advisory Committees 
that would conduct needs assessments in their regions and make recommendations regarding 
technical assistance. The committees said SEAs needed help making better use of scientifically 
based research in decisionmaking, and that strengthening SEAs’ capacity to serve local school 
districts was critical to the success of NCLB reforms, according to a synthesis of their 
recommendations . 1 6 

Following this effort, ED identified states as the Centers’ primary client base, although 
the program had in the past served local clients, as described above. ED charged the Centers to 
work in new ways to expand and strengthen states’ capacity to deliver assistance to schools and 
districts. The new program would also consolidate and expand responsibilities for Center 
assistance in that it would replace the Regional Technology in Education Consortia, the 
Eisenhower National Clearinghouse for Mathematics and Science Education, and the Regional 
Mathematics and Science Education Consortia. 


15 U.S. Department of Education, Office of the Under Secretary, Planning and Evaluation Service, Elementary and 
Secondary Education Division. (2000). Comprehensive Regional Assistance Centers Program: Final Report on the 
Evaluation^ olume I). Washington, DC: Author. 

16 Sheekey, A., Cymrot, D.J., and Fauntleroy, C. (2005, March). A Report to the U.S. Department of Education: 
Overview and Synthesis of the Regional Advisory Committee Reports on Educational Challenges and Technical 
Assistance Needs. Alexandria, VA: The CNA Corporation. 
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Structure of the Center Program 


To implement the provisions of the Educational Technical Assistance Act of 2002, ED 
created a new Center structure with two tiers of technical assistance. The Notice Inviting 
Applications detailed the design. Under the new design, the Secretary of Education would award 
grants to 21 Centers, each tasked with “provid[ing] technical assistance to States as States work 
to help districts and schools to close achievement gaps in core content areas and raise student 
achievement in schools. To accomplish this goal, ED stipulated that applicants had to “propose a 
plan of technical assistance specifically focused on helping States implement the provisions of 
NCLB applicable to States, and helping States build the capacity to help school districts and 
schools implement NCLB provisions and programs.” 17 

While the overall goal of assisting states with NCLB implementation was common to all 
21 Centers, the two-tiered technical assistance designed by ED created distinct roles for the two 
types of Centers. These are described next. 


Distinct RCC and CC Roles 

Within the 21 Centers, ED constituted 16 Regional Comprehensive Centers (RCCs) and 
five Content Centers (CCs). By design, RCCs and CCs were given different roles and functions 
in a system of technical assistance (exhibit 1.1). RCCs, embedded within distinct geographic 
regions across the United States and territories, would deliver technical assistance to the states 
and territories in their region, addressing their needs and building their capacity to assist their 
districts and schools. Meanwhile, each CC would take responsibility for synthesizing knowledge 
from the research and promising practices within a particular substantive area. The CCs would 
build the capacity of the RCCs by providing research-based information, products, guidance, and 
knowledge on key topics. The CCs would also work with RCCs to provide technical assistance 
to states. In turn, each state would help its districts and schools meet NCLB requirements. 


17 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106), 
32585. 
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Exhibit 1.1. 


Center program design 



Elaborating on the functions of RCCs, ED required them to work directly with states to 
“provide frontline assistance.” ED mandated that RCCs provide states with ongoing assistance 
and training that would draw from a range of knowledge sources, including but not limited to 
CCs; provide CCs with information about promising practices; convene states for collaboration; 
and deliver information based on both research and best practice. The specific list of RCC 
responsibilities included the following : 19 

■ “Working closely with each State in its region on an ongoing basis” 

■ “Linking States with the resources of Content Centers, Department staff, Regional 
Educational Laboratories, The What Works Clearinghouse, and other entities” 

■ “Suggesting sources of appropriate service providers or assistance for State 
activities that are not within the core mission of the centers” 

■ “Assisting State efforts to build statewide systems of support for districts and 
schools in need of improvement” 

■ “Working to identify, broker, leverage, and deliver information, resources and 
services from the Content Centers and other sources” 

■ “Convening in partnership with Content Centers and others, as appropriate, States 
and districts to receive training and information on best practices and research- 
based improvement strategies” 

■ “Providing guidance and training on implementation of requirements under 
NCLB and other related Federal programs” 

■ “Facilitating collaboration at the State level to align Federal, State, and district 
school improvement programs” 


18 Ibid.. 32585. 

19 Ibid.. 32585-6. 
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“Helping Content Centers to identify, document, and disseminate emerging 
promising practices” 


The Notice Inviting Applications portrayed CCs as a central source of readily accessible 
knowledge, resources, and tools. Each CC was designed to consolidate in-depth knowledge in 
one of five key content areas: Assessment and Accountability, Instruction, Teacher Quality, 
Innovation and Improvement, or High Schools. The types of knowledge specifically mentioned 
included research, scientifically valid practices, and promising practices. The degree of emphasis 
on research and scientifically valid practice was heightened in this redesign of the Center 
program. This was consistent with NCLB, which stated that scientifically based research must 
inform local practice. 

90 

Within their content areas ED tasked CCs tor 

■ “Identify, organize, select and translate existing key research knowledge. . .and 
communicate the information in ways that that are highly relevant and highly 
useful to State and local level policy makers and practitioners” 

■ “Benchmark State and district practices for implementing NCLB provisions and 
school improvement interventions... and identify promising approaches that can 
be shared with States and districts” 

■ “Convene States and districts, researchers and other experts to leam from each 
other about practical strategies for implementing NCLB provisions and programs” 

■ “Train Regional Center staff on what is known about scientifically valid practices 
and programs” 

■ “Collaborate with Regional Centers to address specific State requests for 
assistance” 

■ “Communicate to the field. . .Department guidance related to the center’s content 
focus” 

■ “Design needs assessment and data analysis tools that States and districts can use 
to benchmark their programs and progress” 


Awards to Centers 

At the conclusion of the competition, new Regional Comprehensive Centers were located 
in 16 regions of the United States, covering all U.S. states and territories, and the five new 
Content Centers had also entered into cooperative agreements with ED (exhibit 1.2). Of the 16 
RCCs, there were four that served only their respective state: New York, Texas, California, and 
Alaska. The remaining 12 Centers served from two to seven states and other jurisdictions. The 
non-state jurisdictions that the Centers were to serve were the following: the District of 
Columbia, Puerto Rico, the Virgin Islands, American Samoa, Commonwealth of the Northern 
Mariana Islands, Federated States of Micronesia [Chuuk, Kosrae, Pohnpei, and Yap], Guam, 
Republic of the Marshall Islands, and Republic of Palau. Throughout this report, the term “state” 
will be defined to include the 50 states as well as these other jurisdictions. A full list of the 
grantees and subgrantees appears in appendix A of this report. 


20 Ibid.. 32586-7. 
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Center funding for each of the three years that are the subjects of this evaluation totaled 
$56.3 million, with variation across Centers (exhibit 1.2). In 2008-09 (FY 2008) individual 
Regional Centers’ funding ranged from a low of $860,000 in Alaska and Pacific to a high of 
$6,039,909 for the California Comprehensive Center. The funding for each RCC was driven by a 
formula based on the region’s total population and its number of poor children ages 5-17. 
Average funding across all RCCs was $2,854, 047. 

Content Center funding for 2008-09 ranged from $1,518,400 for the Assessment and 
Accountability Center and Center on Innovation and Improvement, to $2,518,400 for the Centers 
on Instruction, Teacher Quality, and High Schools. The latter group of Centers was co-funded in 
each year with an allocation of $1 million each from the Special Education Technical Assistance 
and Dissemination Act, authorized under the Individuals with Disabilities Education Act. The 
average funding across CCs was $2,046,096. 


Background on the State Role in NCLB 

A basic premise of the Center program as designed by ED was that NCLB assigned many 
tasks to states. The background section of the Notice itemized the following NCLB requirements 
for states: 

...set standards for student performance, implement statewide testing and accountability 

systems to measure school and student performance toward achieving those standards, 

adopt research-based instructional and program improvements related to teaching and 

learning in the classroom, ensure that all teachers in core subject areas are highly qualified, 

2 1 

and improve or ultimately restructure schools that are consistently low-performing. “ 

With respect to consistently low-performing schools and also low-performing districts, 
NCLB mandated that all states establish and sustain statewide systems of support and 
improvement for school districts and schools identified for improvement under NCLB . Districts 
and schools identified for improvement must receive assistance from support teams, institutions 
of higher education, and regional service centers in the state. This “statewide system of support” 
(SSOS) must include individuals who were knowledgeable about research and practice on 
teaching and learning and who could develop and implement comprehensive improvement 
strategies. “State support teams” (SSTs) were required to help schools plan for improvement and 
to evaluate the effectiveness of school personnel. The NCLB legislation also provided that these 
support teams should receive technical assistance from Comprehensive Centers and others. 22 


21 Ibid.. 32584. 

22 No Child Left Behind. Title I, Part A, § 1 1 17 (a) (1). 
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Exhibit 1 .2. Center funding, by year 


Centers 

States 

FY 2006 

FY 2007 

FY 2008 

Total 


$56,256,713 

$56,256,750 

$56,256,750 

Regional Centers 





Alaska Comprehensive 
Center 

AK 

$850,000 

$860,000 

$860,000 

Appalachia Regional 
Comprehensive Center 

KY, NC, TN, VA, WV 

3,829,927 

3,912,131 

3,912,131 

California Comprehensive 
Center 

CA 

5,912,997 

6,039,909 

6,039,909 

Florida and Islands Regional 

FL, Puerto Rico, Virgin 

3,788,289 

3,869,599 

3,869,599 

Comprehensive Center 

Islands 




Great Lakes East 
Comprehensive Center 

IN, Ml, OH 

3,592,771 

3,669,885 

3,669,885 

Great Lakes West Region 
Comprehensive Center 

IL, Wl 

3,702,196 

2,448,739 

2,448,739 

Mid-Atlantic Comprehensive 
Center 

DE, MD, NJ, PA, DC 

3,388,147 

3,460,868 

3,460,868 

Mid-Continent 
Comprehensive Center 

AR, KS, MO, OK 

2,111,226 

2,156,541 

2,156,541 

New England Comprehensive 
Center 

CT, MA, ME, NH, Rl, VT 

1,644,795 

1,680,099 

1,680,099 

New York Comprehensive 
Center 

NY 

2,886,970 

2,948,935 

2,948,935 

North Central Comprehensive 
Center 

IA, MN, ND, SD, NE 

1,286,458 

1,314,108 

1,314,108 

Northwest Regional 
Comprehensive Center 

ID, MT, WY, WA, OR 

1,630,818 

1,665,821 

1,665,821 

Pacific Comprehensive 

HI, American Samoa, 

850,000 

860,000 

860,000 

Center 

Northern Mariana Islands, 
Federated States of 
Micronesia, Guam, 
Marshall Islands, Palau 




Southeast Comprehensive 
Center 

AL, GA, SC, LA, MS 

4,120,988 

4,209,438 

4,209,438 

Southwest Comprehensive 
Center 

AZ, UT, CO, NV, NM 

2,491,327 

2,544,800 

2,544,800 

Texas Comprehensive Center 

TX 

3,939,324 

4,023,877 

4,023,877 

Content Centers 





Assessment and Accountability Comprehensive Center 

$1,446,096 

$1,518,400 

$1,518,400 

Center on Innovation and Improvement 

1,446,096 

1,518,400 

1,518,400 

Center on Instruction 


2,446,096 

2,518,400 

2,518,400 

National Comprehensive Center for Teacher Quality 

2,446,096 

2,518,400 

2,518,400 

National High School Center 


2,446,096 

2,518,400 

2,518,400 


NOTE: The figure shown in this table for Great Lakes West in FY 2006 is the sum of the six-month "start up" award 
($1 ,243,322) and the FY 2006 award ($2,458,844) as both were awarded in FY 2006. 

SOURCE: U.S. Department of Education 
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Research Questions Addressed in This Report 

The current evaluation takes a global look at the Center program as designed by ED, 
tracking the ways in which the Centers interacted with clients (both states and other Centers) 
over three program years. The first year of data collection was July 2006 through June 2007, the 
second year of program funding. The evaluation was designed to complete its data collection in 
2008-09, covering three of the five program years. 

The priorities for this study focused on the statute’s charge for the evaluation to provide 
“an analysis of the services provided... [and] the extent to which each of the comprehensive 
centers meets the objectives of its respective plan, and whether such services meet the 
educational needs ... in the region.” Among the following key research questions, the first 
focuses on an analysis of services provided and adherence to objectives; the second addresses the 
Centers’ performance in meeting state needs; and the third calls for more detailed assessment of 
the quality, relevance, and usefulness of Center technical assistance: 

1. How did the Regional Comprehensive Centers and Content Centers operate as part of the 

Comprehensive Technical Assistance Center program? 

■ How did Centers develop, refine, and carry out their plans for technical 
assistance? How did they define their clients’ educational needs and priorities? 

■ What were the objectives of the technical assistance the Centers offered? What 
kinds of products and services were provided by the Centers? 

■ How did the Regional Comprehensive Centers and Content Centers coordinate 
their work? 

2. What was the performance of the Comprehensive Centers in addressing state needs and 

priorities? How did their performance change over the period of time studied? 

■ How did the Centers’ state clients define their needs and priorities? 

■ To what extent, as reported by states, did Center assistance expand state capacity 
to address underlying needs and priorities and meet the goals of NCLB? 

■ To what extent did states rely on other sources of technical assistance besides the 
Centers? What were other sources of technical assistance that states used? How 
did the usefulness of Center assistance compare with the usefulness of assistance 
from other sources? 

3. To what extent was the assistance provided by the Centers of high quality, high relevance, 

and high usefulness? 

■ Did the quality, relevance, or usefulness of Center assistance change over the 
period of time studied? 
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■ What was the variation in the quality, relevance, and usefulness of Center 
assistance across types of projects and participants? 

Findings reported here address the Center program as a whole and also address each tier 
of Centers, RCCs and CCs, separately. Data gathered on individual projects were compiled and 
aggregated to the program level and by center type. Program-level aggregated findings reflect an 
average across all sampled projects, and center-type aggregated findings reflect the average 
across all RCC sampled projects or all CC sampled projects. Additional details are provided in 
chapter 2. 

Organization of This Report 

Following this introductory chapter, chapter 2 describes the study’s methods. Chapter 3 
addresses the first research question, describing Center technical assistance, procedures for needs 
assessment, and the functioning of the two-tiers of technical assistance. Chapter 4 addresses the 
second research question, describing the technical assistance priorities reported at the state level 
and the ways in which Centers were reported to have addressed these priorities and expanded 
state capacity. Chapter 5 addresses the third research question, reporting on the quality, 
relevance, and usefulness of selected Center projects as determined through expert review (for 
quality) and participant surveys (for relevance and usefulness). The report concludes with several 
appendices that provide additional technical notes, the materials used to collect data (surveys, 
protocols for Center interviews, and requests for materials from the Centers), and historical 
exhibits. 
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2. Study Design 


The evaluation team used six data sources in each of the three rounds of data collection to 
address the report’s research questions: documents produced by the Centers with assistance from 
the evaluation team, Center management plans, interviews with Centers, state manager surveys, 
expert panel reviews, and participant surveys (exhibit 2.1). Data were collected from all sources 
for the 2006-07, 2007-08, and 2008-09 program years to describe changes over time in the 
Center operations and assessments of Center’s technical assistance. This chapter describes those 
data sources and analytic procedures used in the analysis. 

For the first three questions, which pertained to the operations of the Centers, the data 
were drawn from the Centers themselves. The team used Center management plans as a data 
source regarding Center objectives. In the summers of 2007 and 2008 as well as spring 2010, the 
team conducted interviews with the Centers; closed-ended prompts were used in face-to-face 
interviews to gather self-report data systematically on Center objectives, procedures for needs 
assessment and planning, and interactions with their clients. The final interviews (2008-09) 
included structured questions to gather Center perspectives on changes in technical assistance 
work across program years and on quality assurance procedures. Two documentary sources were 
used as sources for descriptions of Center technical assistance: the project inventory forms and 
project cover sheets completed by the Centers with review and feedback from the evaluation 
team. The definition of a project and the procedures for gathering and reviewing these data are 
discussed in detail in this chapter. 

A survey of state managers was the source for data on the states’ technical assistance 
needs and priorities, on other sources of technical assistance used, ratings of the overall technical 
assistance received, and on perceived capacity change at the state level. The technical quality of 
Center technical assistance was assessed by a panel of experts on the topic of each technical 
assistance project. Finally, participants answered survey questions pertinent to the relevance and 
usefulness of Center technical assistance. 

Each of these data sources is described in this chapter, and the analytic procedures 
specific to each source are discussed. The data collection instruments and further details 
regarding data sources and procedures can be found in appendices referenced throughout the 
chapter. The chapter concludes with a brief explanation of units of analysis and use of statistical 
tests. 
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Exhibit 2.1. 


Data sources for the research questions, 2006-07, 2007-08, and 2008-09 


Research question 

Project 

inventory forms 
and project 
cover sheets 

Center 
manage- 
ment plans 

Inter- 
views with 
Centers 

State 

manager 

survey 

Expert 

review 

panels 

Participant 

surveys 

How did the Regional Comprehensive Centers and Content Centers operate as part of the Comprehensive Technical Assistance Center program? 

How did Centers develop, refine, and carry out their plans for 
technical assistance? How did they define their clients’ 
educational needs and priorities? 




✓ 


✓ 

What were the objectives of the technical assistance the 
Centers offered? What kinds of products and services were 
provided by the Centers? 



■/ 




How did the Regional Comprehensive Centers and Content 
Centers coordinate their work? 








What was the performance of the Comprehensive Centers in addressing state needs and priorities? How did their performance change over the period 
of time studied? 


How did the Centers’ state clients define their needs and , 

priorities? 

To what extent, as reported by states, did Center assistance 

expand state capacity to address underlying needs and S 

priorities and meet the goals of NCLB? 

To what extent did states rely on other sources of technical 
assistance besides the Centers? What were other sources of 

technical assistance that states used? How did the S 

usefulness of Center assistance compare with the 
usefulness of assistance from other sources? 


To what extent was the assistance provided by the Centers of high quality, high relevance, and high usefulness 


Did the quality, relevance, or usefulness of Center 
assistance change over the period of time studied? 


What was the variation in the quality, relevance, and 
usefulness of Center assistance across types of projects and 
participants? 


✓ 

✓ 

✓ 

(overall 
relevance and 

(quality) 

(project-level 
relevance and 

usefulness) 

✓ 

✓ 

usefulness) 

✓ 

(overall 
relevance and 

(quality) 

(project-level 
relevance and 

usefulness) 


usefulness) 



Identification of a Sample of Center Work for Expert Review 
and Participant Surveys 


A critical component of the evaluation was to rate the quality, relevance, and usefulness 
of Center products and services. Given available resources, it was not possible for the evaluation 
team to submit all of a Center’s products and services to an independent review panel to rate 
quality. Nor was it feasible to survey all individuals who used Center products or participated in 
Center activities in the designated time period regarding relevance and usefulness. Therefore, the 
evaluation team developed and applied a strategy to select a sample of work from each Center 
for expert panel review and participant ratings. 

The following sections describe: (1) the unit of analysis, (2) the sample frame, (3) the 
evaluation sample, and (4) materials obtained from Centers. 


The Unit of Analysis: The Project 

The evaluation team initially reviewed the Centers’ 2006-07 management plans to 
understand the nature of the work the Centers were conducting and determine if the management 
plans might serve as an appropriate sampling frame for the evaluation. Based on this effort, the 
team determined that the sampling procedures could not be based on the management plans for 
two reasons. First, the Centers’ plans and work continued to evolve over time. Consequently, the 
plans, which were prepared before the program year, did not comprehensively reflect the work 
actually being done by the Centers months later. Second, the Centers used different approaches 
to organizing and aggregating their work. For example, some presented their work by state while 
others organized it by topic area. For the purposes of this evaluation, the team identified 
“projects” as a common level of aggregation that would constitute units large enough for review 
and rating, but focused enough for coherence. A “project” was defined as a group of closely 
related activities and/or deliverables designed to achieve a specific outcome for a specific 
audience. To ensure that projects would constitute units that were large enough for review and 
rating, but focused enough for coherence, the study team provided the following criteria: 

■ Complete and coherent whole. Because each project should be able to stand on 
its own in an expert panel review, it should include all related activities and 
products. 

■ Common intended outcome. Where a cluster of activities and deliverables was 
designed by the Center to lead to the same outcome for the same audience, those 
activities and deliverables should be grouped as one project. 

■ Topical focus. With few exceptions, a project addressed just one topic (for 
example, effective systems of support, adolescent literacy, assessment of English 
language learners) around which there was a body of research or professional 
wisdom. 
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Since the project was a unit developed for this evaluation and was not necessarily how 
Centers divided up their work for programmatic or cost-tracking purposes, the evaluation team 
took several steps to ensure that the project concept was clear and used consistently across 
Centers. The evaluation team developed a standard Project Inventory Form (PIF) that Centers 
used to create an inventory of their work for each of the program years in this evaluation (see 
appendix B for a copy of the PIF) as well as written guidance in the instructions for completing 
the PIF (described in appendix B) and a sample inventory to serve as an example of the kinds of 
projects that should be listed by the Centers. The sample inventory included examples of 
activities or resources for defining projects at each level of effort — major, moderate, or minor. 
Members of the study team also invited Center staff to attend training sessions by conference call 
on completing the PIF, including defining projects. When draft inventories were received, the 
study team reviewed them to make sure the entries listed met the three criteria used to identify 
projects. They also reviewed the Center’s reporting of the project level of effort for 
correspondence with the listed activities and resources, and listing of the projects under each 
topic for correspondence with the topic definitions provided in the written guidance. The study 
team provided technical assistance to Centers as needed to ensure that all projects conformed to 
the standards. 


The Sampling Frame 

The evaluation was designed to assess the quality, relevance, and usefulness of a sample 
of Center work. The PIFs served as the basis for identifying the sampling frame from which the 
study team drew the sample of projects that became the subject of expert panel reviews and 
participant surveys. For each data collection cycle, the evaluation team asked each Center to use 
the PIF to prepare an inventory of all the projects active during the appropriate grant period. 23 
Exhibit 2.2 shows the total number of projects on the PIFs by project size for each of the data 
collection cycles.' Projects that spanned multiple years were counted each year they appeared 
on the PIF; therefore, the total number of distinct projects is less than the sum total of individual 
projects listed in Exhibit 2.2. 


23 In the instructions provided to Centers for completing the PIFs, “projects” excluded the following activities: (1) 
training or professional development of Center staff, (2) work on coordinating committees within the 
Comprehensive Center network, (3) annual needs assessments or negotiations with states, unrelated to specific 
projects, and (4) other internal meetings or documents. 

24 Centers designated on their PIFs whether each project was “major,” “moderate,” or “minor" in terms of the level 
of effort and/or resources the Center devoted to it, relative to other projects in the same Center. Examples of 
activities or resources defining each project level of effort were provided in the instructions for the PIFs. After draft 
inventories were received, the evaluation team also reviewed the Center’s reporting of the project level of effort for 
correspondence with the listed activities and resources and followed up with the Centers where there were questions. 
Although instructions were provided with examples and checks were conducted, these designations were not 
standardized across Centers and as a result a project considered major by one center might be viewed as moderate or 
minor by another Center. 
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Exhibit 2.2. Number of projects on the project inventory forms (PIFs), by 
project size and by year 


Project size 

2006-07 

2007-08 

2008-09 

Total 

364 

346 

331 

Major 

110 

111 

108 

Moderate 

106 

111 

110 

Minor 

148 

124 

113 


EXHIBIT READS: There were 364 projects included on the PIFs for the 2006-07 
program year. Of these, 110 were classified as major projects, 106 were 
moderate projects and 148 were minor projects. 

SOURCE: Project inventory forms. 

To be eligible for the sampling frame, projects first needed to represent a reasonable 
amount of effort (i.e., classified as “major” or “moderate” by the Center) and have a sufficient 
amount of material to give reviewers enough information to judge the quality of the work. 
Projects included in the sampling frame also needed to have identifiable participants since the 
evaluation design called for collecting relevance and usefulness ratings through surveys of 
project participants. All minor projects (approximately one-third of the Center projects in each 
year) were excluded from the sampling frame, as few of them had identifiable participants or 
sufficient materials for panel review. Thus, the sampling frame represents the portion of each 
Center’s work that they identified as using a major or moderate amount of their efforts or 
resources. 


The Sample of Projects 

To ensure that the final sample of projects reviewed each year reflected a range of each 
Center’s work across topics and states and captured work that represented the largest investment 
of resources, the team implemented a sampling strategy that included a combination of Center- 
nominated projects and a stratified set of purposively-selected projects. 25 The desired sample 
size at each Center was a function of that Center’s budget amount. For each of the three data 
collection years, the number of projects sampled for each Center ranged from three to eight 
depending on the size of the Center’s annual budget. Centers with smaller budgets (less than 
$2 million) were asked to nominate one project for inclusion in the study sample while those 
with budgets of $2 million or more were asked to nominate two projects. Exhibit 2.3 shows the 
target and actual sample distribution by budget size across study years. In some cases, the 
Centers did not have enough eligible projects in the sample frame to meet their desired sample 
size so the total number of projects sampled was lower than expected. Although the sample was 
not statistically representative in scientific terms, it was designed to include a high percentage of 
the major projects of each Center as well as projects that Centers thought best represented their 
work. 


25 The full list of decision rules, applied across all Centers sampling projects within center are provided in appendix 
B. 
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To increase buy-in from the Centers and to allow them to showcase a project of their 
choice, Centers were given an opportunity to nominate the one or two projects they felt best 
represented their work. 26 These Center- nominated projects were selected first for the sample. 
Selection of the remainder of the sample was then completed by the evaluation team using an 
iterative sampling process to randomly select projects while controlling for topic and state. Major 
projects (across topics and states) were sampled first and then moderate projects (if needed). 
When choosing between projects of similar categorical size, the evaluation team selected 
projects from different topics before sampling multiple projects within a given topic. When 
choosing between projects of similar categorical size and topic, the evaluation team selected 

27 

projects from different states before sampling multiple projects within a given state.” 

Using these methods, the evaluation team selected a total sample of 122 projects in 2006- 
07 and 2007-08 and 118 projects in 2008-09. Exhibit 2.4 shows the number of projects in the 
sample by project size and by year. In each of the years, the sample predominantly included the 
most major work of the Centers although the sample was not statistically representative in 
scientific terms, and covered 33 to 36 percent of all projects reported on the PIFs. 


~ 6 The number of projects a Center was able to nominate depended on the size of its annual budget, as shown in 
exhibit 2.3. 

27 Overall, the procedures used to select the sample of projects to be included in the study each year were intended to 
provide a “fair” representation of Center work. In addition to the bias introduced by Center nominations of projects 
they considered to best represent their work, there are two dimensions for which there is the potential for selection 
bias in the sample of projects. Projects were selected to ensure inclusion in the study sample of projects covering a 
variety of content areas and serving all geographic areas. However, this strategy produced a non-probability sample. 
In addition, this strategy ensured that topic areas addressed by the largest number of projects are underrepresented in 
each year’s project sample, and topic areas addressed by relatively few projects are overrepresented. It is also 
possible that topic areas addressed by relatively few Centers would be underrepresented in the sample. The other 
dimension on which the sample is likely to be biased is project size. The evaluation team deliberately selected major 
projects over moderate projects whenever possible. To the degree that the major projects are unrepresentative of the 
Centers’ work conducted under moderate or minor projects, then the study sample is biased. 
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Exhibit 2.3. Sample size by budget 



Target 

Number of 
projects 

Number of projects selected 
across Centers in: 

Center funding 
level — annual federal 
contract 

(number of Centers) 

number of 
projects to be 
selected for 
review, per 
Center 

nominated/ 
Number of 
projects selected 
purposively, per 
Center 

2006-07 

2007-08 2008-09 

Total 



122* 

122 

118 

Less than $1 million 
(N=2) 

4 

1/3 

7 

6 

7 

$1 to 1 .9 million 
(N=5) 

5 

1/4 

25 

23 

25 

$2 to 2.9 million 
(N=7) 

6 

2/4 

41 

43** 

40 

$3 to 3.9 million 
(N=5) 

7 

2/5 

34 

34 

31 

$4 million or more 
(N=2) 

8 

2/6 

16 

16 

16 


* This total includes a collaborative project conducted jointly by an RCC and a CC that is accounted for once in the 
total sample but included once for each relevant Center in the table rows. 


** The total number of projects exceeded the expected target by one due to a program that was misclassified in a 
budget category with a larger target sample size. 

EXHIBIT READS: There were two Centers whose annual funding level was less than $1 million. Four projects were 
selected for review for each of these centers, one of which was nominated by the Center for selection and three of 
which were selected for inclusion by the study team. 

SOURCE: U.S. Department of Education. 


Exhibit 2.4. Number of projects in the study sample, by project size and 
by year 


Project size 

2006-07 

2007-08 

2008-09 

Total 

122 

122 

118 

Major 

93 

88 

92 

Moderate 

29 

34 

26 


EXHIBIT READS: There were 122 projects in the project sample for the 2006-07 program year. Of these, 93 
were classified as major projects and 29 were classified as moderate projects. 

SOURCE: Project inventory forms (PIFs) submitted by the Centers. 

Further detail on the sampled projects by topic in relation to all major and moderate 
Center projects for the 2008-09 data collection cycle appears in exhibit 2.5. Refer to appendix B, 
exhibits B.2 and B.3 for similar exhibits showing the sampled projects by topic for 2006-07 and 
2007-08. In 2008-09, 50 percent or more of all major or moderate projects were selected for the 
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project sample in 16 of the 23 28 topics. This figure was 17 of 23 for 2006-07 and 15 of 23 for 
2007-08. 

Exhibit 2.5. Distribution of all major and moderate projects and projects 
in the evaluation sample, by topic in 2008-09 



Number of 


Percent of 


major and 

Number of 

all major and 


moderate 

projects in the 

moderate 


projects on 

sample 

projects in 


project 

(all major or 

evaluation 

Project topic 

inventories 

moderate) 

sample 

Total 

218 

118 

54% 

Components of effective systems of 
support — state, district, school 

54 

33 

61 

Data use/data-driven decisionmaking 

6 

3 

50 

Formative assessment 

4 

2 

50 

Reading 

6 

3 

50 

Adolescent literacy 

10 

6 

60 

Mathematics 

16 

8 

50 

Dropout prevention 

4 

2 

50 

High school redesign/reform 

7 

4 

57 

Transition to high school 

2 

2 

100 

Special education — curriculum, instruction, 

-j 

n 

n 

and professional development 

1 

u 

u 

Special education — assessment 

5 

i 

20 

English language learners 

26 

12 

46 

Highly qualified teacher provisions of 

-| -| 

4 

36 

NCLB 



Teacher preparation and induction 

5 

2 

40 

Teacher professional development 

4 

4 

100 

Supplemental educational services (SES) 

2 

2 

100 

Response to Intervention (Rtl) 

14 

9 

64 

Migrant education 

7 

3 

43 

Indian/native American education 

1 

1 

100 

Data management compliance 

9 

5 

56 

Assessment design 

3 

2 

67 

Parent involvement 

6 

2 

33 

Other 

15 

8 

53 


EXHIBIT READS: For the 2008-09 program year, there were 54 major and moderate projects on the project 
inventories that focused on components of effective systems of support. Of these, 33 (or 61 percent) were in the 
project sample. 

SOURCE: Project inventory forms submitted by the Centers. 


28 In all years, the topic area of “other” is included as the 23 ld topic area, as a small number of the sampled projects 
were included in that area. Subsequent discussions in the report use the 22 specific substantive topic areas. 
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Materials from Centers 


The evaluation team notified the 21 Center Directors about projects selected for review 
and sent a standard Request for Materials for Expert Panel Review (see appendix B for a copy of 
the transmittal memo and request form). Centers were asked to assemble and submit a 
comprehensive set of pre-existing materials associated with each project (meeting agendas, 
briefing books, meeting summaries, training materials, white papers, web resources, etc.) that 
would fully describe the project and provide reviewers with a sufficient basis for rating the 
technical quality of the work. Centers were also asked to include a participant list and a standard 
cover sheet, using a format developed by the evaluation team, for each project. The participant 
lists were used to draw the sample for participant surveys. 

When the project materials were received, they were reviewed by evaluation team 
members for completeness. If materials were missing or inaccessible (e.g., electronic files didn’t 
open) or the cover sheet was incomplete, a follow-up memo was sent to the Center describing 
any issues. Evaluation team members then worked with the Center to obtain the missing 
information and finalize the review package (see appendix B for a copy of the follow-up memo). 


Describing Center Operations 

The purpose of collecting data from the Centers was to describe Center operations and to 
address research questions regarding the objectives of the Centers, the kinds of products and 
services provided, how the Centers defined their clients’ needs and priorities, and the extent to 
which the Centers met the objectives of their own plans. The description of Center operations in 
chapter 3 of this report is drawn from three sources of data: interview responses gathered from 
the Centers, PIFs, and the cover sheets submitted for the sampled projects. The procedures used 
in gathering and analyzing data from these sources in all three years are described in the 
following pages. 


Management Plans 

Centers stated their objectives for each year in their annual program management plans. 
For example, for the 2006-07 program year, Centers had developed their management plans in 
spring 2006, prior to the program year. The evaluation team relied on these management plans to 
gather information from statements of the Center’s intended substantive focus for the year’s 
technical assistance. The team coded each Center’s stated objectives in their management plan 
by subject area, using the same list of 22 topics used to code the PIFs. Appendix C provides a 
description of the process used to code the management plans as well as an analysis of the 
intercoder reliability. 
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Interviews with Centers 


The evaluation team visited each of the Centers in person in the summers of 2007 and 
2008 and conducted the third interview by phone in spring 2010. The primary purpose of these 
interviews was to capture descriptions of Center operations during each of the program years. 
Interviews conducted with Center directors and other key leadership staff posed a combination of 
structured open-ended and binary questions regarding the following topics (see appendix C for 
the protocols): 


■ Center organization (lead organization, subgrantees, ways of dividing 
responsibilities among staff) 

■ Major areas of focus 

■ Communication with client organizations (states in the case of RCCs, or RCCs 
and states in the case of CCs) regarding needs and assistance to be provided 

■ Modes of delivering technical assistance 

■ Reasons for not carrying out technical assistance requests 

■ Barriers to delivering technical assistance 

■ Approaches taken in quality assurance 

■ Working relationships within the Center network 

For the third round of Center interviews, the evaluation team added probes on topics of 
emerging policy interest such as the ways in which Centers viewed their work to have evolved 
over the years, the Centers’ views on barriers to providing technical assistance, and additional 
description of the process for quality assurance that Centers established. 


Project Inventory Forms (PIFs) 

While the main purpose of the PIFs was to build the sample frame (described earlier in 
this chapter), the evaluation team also used the PIFs to gain data on the work the Centers 
undertook in each of the study years. The projects sampled for quality, relevance, and usefulness 
ratings were classified by topic (see appendix B). 


Project Cover Sheets 

For projects included in the sample each year, the standard cover sheets provided by the 
Centers described project activities and cross-Center collaboration in the project. The project 
cover sheets were primarily collected from each Center to help orient expert panels to the 
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purpose and content of the materials to be reviewed (see appendix B for a copy of the Request 
for Materials for Expert Panel Review transmittal memo and cover sheet). 

The evaluation team used the cover sheets as a data source for an overall description of 
activities and collaboration in the sampled projects (reported in chapter 3) and for use in 
classifying projects into subgroups for which the ratings of relevance and usefulness could be 
analyzed (reported in chapter 5, as described below). The cover sheets provided descriptive 
information for each project, including the activities and deliverables associated with the project 
and the contributions of other Centers to the project. Categories of Center activities and 
resources were drawn from review of the Center management plans and site visit interviews. 
These coding categories, thus, permitted a yes/no judgment of whether each project offered each 
of the following activities or resources to participants: 

■ Ongoing consultation and follow-up 

■ Research collections or syntheses 

■ Engagement of participants in project planning 

■ Training events 

■ Task force meetings and work 

■ Conferences 

■ Support for development of a formal plan to implement a program or policy 

The team also coded the type of contribution, if any, of any CC to each RCC project, and 
of any RCC to each CC project. 

Additionally, for the 2008-09 data collection cycle, the evaluation team used the project 
cover sheets to provide additional detail regarding the 33 projects in the study sample included 
under the topic of effective systems of support. For these projects, the evaluation team coded 
activities and resources at a more fine-grained level. 

All these elements of the cover sheets were coded by members of the evaluation team 
using procedures described in appendix C; the appendix also provides a detailed description of 
all the codes used as well as the results of the analysis of intercoder reliability. 


Survey of State Managers 

The purposes of the survey of state managers were to obtain information on state 
priorities in terms of state responsibilities related to the implementation of NCLB, obtain the 
state perspective from SEA administrators working with the Centers on the relevance and 
usefulness of Center assistance, and obtain a comparative judgment of Center assistance in 
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relation to assistance available through other sources such as professional associations. The 
survey instrument used for the survey of state managers appears in appendix E of this report. 

In order to identify appropriate respondents for the survey of state managers, each year 
the evaluation team collected the names of each RCC’s main point(s) of contact in each SEA 
from the RCCs. For each of the data collection years, this resulted in a total of over 120 

29 

respondents across the 62 states included in the study. 

To be fully reflective of all SEAs in the analysis of these data, it was critical that the 
study team receive completed surveys from state managers in each state. In 2006-07 there was at 
least one response from each of the 50 states and 6 of 12 outlying areas, in and 2007-08 there 
was at least one response from each of the 50 states and 7 of 12 outlying areas. In 2008-09, the 
evaluation team received responses from 48 states and 6 of 12 outlying areas. 

In each of the years, there were a number of states for which the evaluation team received 
completed responses from more than one state manager. Exhibit 2.6 shows the number of 
responses to the state manager survey for each of the years. The state was the primary unit of 
analysis in analyzing data from the state manager survey for this report. The state managers’ 
responses were weighted to ensure that each state was equally represented in all summary 
statistics while taking into account the variation in responses within each state. The weighting 
procedure, where each response was weighted by the inverse of the number of managers 
responding from that state, ensured that each state was equally represented when the evaluation 
team aggregated responses across states to describe the distribution of responses. 


Exhibit 2.6. Number of responses to the state manager survey, by year 


Number of responses 


Number of states 


from the states 

2006-07 

2007-08 

2008-09 

Total 

56 

57 

54 

1 

36 

30 

34 

2 

15 

21 

19 

3 

4 

5 

1 

4 

1 

1 

0 


EXHIBIT READS: For the 2006-07 program year, in 36 states the survey of state 
managers was completed by a single respondent. There was one state where the state 
manager survey was completed by four separate respondents. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was 
equally represented in instances where more than one manager from a state responded. 


29 The Centers were expected to serve 62 jurisdictions including the 50 states and 12 other jurisdictions: the District 
of Columbia, Puerto Rico, the Virgin Islands, American Samoa, Commonwealth of Northern Marianas, Federated 
States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap), Guam, Republic of the Marshall Islands, and Republic of 
Palau. For purposes of this report, the term “state” refers to the 50 states and the outlying territories listed here. 
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At the conclusion of the state manager survey, respondents were asked to describe the 
offices or departments within their state department of education they directed. The responses to 
this question show that across the states the offices most often represented by the state managers 
were federal programs and school improvement. Exhibit 2.7 shows the distribution of weighted 
responses to these questions across the three data collection years. 


Exhibit 2.7. Office division or department directed by state manager survey 
respondents, by year 


Percent of state managers (weighted) 


Office/department 2006-07 2007-08 2008-09 


Federal programs 

68% 

64% 

65% 

School improvement 

64 

66 

67 

Curriculum and 
instruction 

53 

50 

45 

Assessment and 
accountability 

40 

45 

38 

Special education 

30 

28 

24 


NOTE: Survey respondents were able to select multiple responses to this question and as 
a result, the percentages do not add to 100 percent. 

EXHIBIT READS: For the 2006-07 program year, 68 percent of state managers 
(weighted) indicated that they directed the federal programs office in their state. Sixty-four 
percent of the state managers (weighted) indicated that they headed the school 
improvement office. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was 
equally represented in instances where more than one manager from a state responded. 


Assessment of Quality by Expert Panels 

Given the Centers’ charge to focus on scientifically based research and evidence-based 
practices to improve student achievement and close achievement gaps, one goal of the evaluation 
is to assess the technical quality of work across the Centers using an independent panel of expert 
reviewers. Each sampled project was independently rated by a panel of three experts. 30 
Reviewers assigned a score to each of three quality dimensions discussed below, using a 5-point 
rating scale. 

In an effort to maximize interrater reliability, the evaluation team: (1) defined quality and 
developed a detailed scoring rubric that could be applied across all Centers and a range of 
projects; (2) recruited, trained and assigned highly qualified expert panelists; and 
(3) implemented a process for reviewers to discuss their findings with one another when scores 
were discrepant for particular projects. The following sections describe each of these steps, 
followed by a brief discussion of how the final technical quality ratings were calculated. 


30 


In each of the years, there were between 2 and 9 projects that were rated by only two panelists. 
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Define Technical Quality and Develop Scoring Rubric 

Developing the definition of technical quality and the rubric used to measure it 
incorporated information from many sources, including federal legislation and ED specifications 

•5 1 

on what constituted scientifically based evidence and an example from another federal agency. 
When defining technical quality and developing the scoring rubric, the evaluation team sought to 
ensure that the definition was relevant to the range of projects Centers would provide- 
recognizing that some projects might have a substantial research knowledge base and others 
might be guided more by promising practices (those that were supported by evidence but not yet 
rigorously studied) or legislative or regulatory requirements. Also, the definition of quality and 
the associated rubric had to be applicable to projects at varying points of development and 
implementation, from early-stage needs assessment and design work to fully-developed products 
and services. The evaluation team developed a quality scoring rubric, included in appendix G, to 
assess quality along the following three dimensions: 

■ Dimension 1: Demonstrated use of the appropriate documented knowledge base 

■ Dimension 2: Fidelity of application of the knowledge base to the products and 
services provided 

■ Dimension 3: Clear and effective delivery 

Reviewers assigned a score to each dimension, using a 5-point rating scale (where 1 
meant “very low quality” and 5 meant “very high quality”), according to the indicators defined 
for each dimension and examples in the scoring booklet. 


Recruit, Train, and Assign Expert Panelists 

To meet the selection criteria for this evaluation, expert panelists had to have current, 
rigorous work in the particular topic of interest (for example, publications in peer-reviewed, 
scholarly journals; presentations at relevant professional organization meetings; recent 
membership on advisory panels or task forces) and be free of conflicts of interest. Nominations 
for panelists were made by staff in ED’s Institute of Education Sciences (IES), the Technical 
Work Group for this evaluation, members of the evaluation team, and Center staff. Based on the 
selection criteria, for the 2006-07 data collection cycle the evaluation team selected a total of 70 
expert panelists, 94 percent with a doctorate degree and two-thirds (67 percent) with university 
affiliations, to review the Centers’ 2006-07 sampled projects. 33 An additional 14 expert 
reviewers were recruited and trained for the second round of reviews to replace four reviewers 


31 Review of Instructional Materials for Middle School Science. (1997, February). National Science Foundation, 
Retrieved from http://www.nsf. gov/pubs/1997/nsf9754/nsf9754.htm?org=NSF . 

32 Current Comprehensive Center staff, as well as individuals employed by organizations that had an ongoing 
financial relationship (for example, a contract or cooperative agreement) with a Comprehensive Center and who 
worked on a Center project, were not eligible to serve as reviewers. 

33 The final expert panels for 2006-07 were made up of 67 expert reviewers. One of the initial trainees was removed 
due to a conflict of interest, and two reviewers were later dropped from the review process when they failed to 
complete their reviews in a timely fashion. 
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who did not return and to add additional depth in certain topical areas, in particular state systems 
of support. All but two of the 77 second round reviewers returned for the third and final round of 
reviews. 

The evaluation team sought to maximize interrater agreement in scoring of quality by 
training expert reviewers to systematically use a standard rubric. During the two-day training, 
experts worked in small groups to discuss how each of the three quality dimensions, and their 
corresponding indicators, applied to sample project descriptions that were provided by the study 
team. Reviewers independently scored one of two projects overnight and submitted their 
dimension-level scores at the beginning of the second day. Scores were posted and analyzed in 
terms of rater agreement at the dimension level. The panelists again worked in small groups to 
discuss their scores and identify possible reasons for any discrepancies in the results. During the 
small group discussions, evaluation staff circulated among the groups to assess whether 
reviewers had adhered to the standards of evidence discussed on Day l. 34 The same training was 
offered to newly recruited reviewers for the second round of reviews, while returning reviewers 
completed a refresher training session offered by Webinar. A similar refresher training session 
was also provided for all returning reviewers in advance of the third and final round of reviews. 

During the actual review process, expert reviewers were asked to score four to eight 
projects in their area(s) of expertise, with no more than three projects from any given Center. 
Since judgments about the state of the available evidence on a given topic and its applicability to 
the project being rated relied heavily upon the knowledge of the expert reviewers, it was 
important that reviewers were assigned projects that matched their area(s) of expertise. The 
evaluation team was also careful when assigning projects to avoid known conflicts of interest. 
When needed, the evaluation team reassigned projects when reviewers were unable to complete 
their reviews in a timely fashion, identified unforeseen conflicts of interest, or did not feel they 
had the requisite expertise to review the assigned projects. 

Overall, between 93 and 98 percent of all projects sampled were reviewed by three 
panelists. In each year, a small number of projects (9 projects for 2006-07, 2 projects for 2007- 
OS, and 3 projects for 2008-09) were reviewed by only two panelists because the third assigned 
reviewer did not complete the review within a reasonable timeframe. 36 


34 During training, expert reviewers scored a sample project and discussed their scores in small groups, paying 
particular attention to areas where scores were divergent. This approach allowed the evaluation team to identify 
dimensions or indicators within the scoring rubric that seemed to be problematic (resulting in discrepant scores) or 
particular reviewers who needed additional training or appeared to be inappropriate to use. The goal was to have 
panelists leave the training with a common understanding of how to apply the quality rubric; the evaluation team did 
not intend to attempt to establish a specific interrater reliability criterion at training. 

35 For the purpose of assignments, the term “conflict of interest” meant any financial or other interest that appeared 
to conflict with or significantly compromise the service of the individual reviewer because it could significantly 
impair the individual’s objectivity. 

3h In consultation with IES, the evaluation team decided to forgo the third review on these particular projects since 
there was interrater agreement (i.e., a difference less than 2 points) between the two scores that had been submitted 
by the other panelists assigned to these projects. 
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Address Interrater Reliability 

In addition to the detailed scoring rubric and training provided, a resolution process was 
used to help achieve a high degree of interrater reliability in scoring. If the reviewers’ project- 
level scores (defined as the simple average of their three dimension-level scores) were found to 
differ by 2 or more points from each other for any given project, the evaluation team convened 
the panel by telephone to discuss the ratings. The goal of the discrepancy conference calls was 
to give panelists an opportunity to understand the rationale behind their colleagues’ scores and 
consider whether, on the basis of that discussion, any scoring revisions were warranted. The 
evaluation team emphasized to the reviewers that these discussions were not intended to achieve 
consensus among them. 

Overall, between 76 and 83 percent of them had no discrepancies in the original project- 
level scores (exhibit 2.8). After a discrepancy conference call, panelists were given the 
opportunity to submit a revised scoring form and narrative report of the project’s strengths and 
weaknesses. Although panels were not required to reach consensus, discrepancies were resolved 
in all but two to four cases, increasing the number of projects in agreement from 76 percent to 98 
percent in 2006-07, and from 83 percent to 97 percent in 2007-08 and 2008-09. 

IQ 

To estimate interrater agreement, we calculated the rWG(J) index. Before the 
discrepancy calls, the average rWG(J) coefficient ranged from 0.75 to 0.87 across the three years 
of review (exhibit 2.9). After the calls were conducted, the value increased to 0.87 for program 
year 2006-07 and 0.88 for 2007-08 and 2008-09. 


7 The 2-point difference threshold was selected in an effort to identify projects with widely divergent scores that cut 
across qualitative categories (i.e., “low” quality versus “high" quality), while at the same time minimizing burden on 
expert panelists. 

3S James, L.R, Demaree R.G., and Wolf G. ( 1984). Estimating within-group interrater reliability with and without 
response bias. Journal of Applied Psychology , 69(1), 85-98. 

39 We estimated interrater agreement using the rWG(J) index. Treating each dimension as an “item,” we first 
calculated the score variance between raters on each dimension and then averaged the three variances. We assumed 
the random error variance to be 2.0 for a 5-point scale, as suggested by James et al. (1984). An rWG value between 
0.71 and 0.90 is generally considered an indicator of “strong agreement” (LeBreton, J.M., and Senter, J.L. (2008, 
October). Answers to 20 questions about interrelated reliability and interrater agreement. Organizational Research 
Methods , 11, 815-852. Retrieved from http://orm.sagepub.eom/cgi/content/abstract/ll/4/815 ). 
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Exhibit 2.8. Distribution of discrepancies and results of the discrepancy 
call process, by year 


Description 

2006-07 

2007-08 

2008-09 

Total number of projects 
reviewed by expert panels 

122 

121 

118 

Projects without a project- 
level discrepancy 

93 

(76 percent of 1 22) 

100 

(83 percent of 121) 

98 

(83 percent of 1 18) 

Project-level discrepancies 
identified 

29 

(24 percent of 1 22) 

21 

(17 percent of 121) 

20 

(17 percent of 1 18) 

Project-level discrepancies 
resolved* by panel calls 

27 

(93 percent of 29 
calls conducted) 

17 

(81 percent of 21 
calls conducted) 

16 

(80 percent of 20 
calls conducted) 

Project-level discrepancies 
remaining after panel calls 

2 

(7 percent of 29 
calls conducted) 

4 

(19 percent of 21 
calls conducted) 

4 

(20 percent of 20 
calls conducted) 


* Discrepancies were considered resolved when the project-level scores differed by less than 2 points. 

EXHIBIT READS: Of the 122 projects reviewed for the 2006-07 program year, 93 (76 percent) had no discrepancies 
in the project level scores. 


Exhibit 2.9. Interrater agreement before and after discrepancy calls (original 
versus revised scores), as measured by the rWG(J) index, by year 


rWG(J) index 

2006-07 

2007-08 

2008-09 

Before discrepancy calls 
(original scores) 

0.75 

0.81 

0.87 

After discrepancy calls 
(revised scores) 

0.87 

0.88 

0.88 


EXHIBIT READS: The interrater agreement index for the original scores for 2006-07 
project sample was 0.75. This number increased to 0.87 after discrepancy calls were 
conducted. 


Calculate Aggregate and Dimension-Level Measures of Technical Quality 

In order to analyze the technical quality of sampled projects in each year, a series of steps 
was taken to combine individual expert ratings into aggregate scores. The first step was to 
combine the individual dimension-level scores into a single rating from each panelist. For each 
project reviewed, a simple average of the three dimension-level scores was computed to generate 
a project-level score for each reviewer. In the second step, the three reviewers’ scores were 
averaged to determine the overall quality score for each project. In the third step, the overall 
project scores across the set of sample projects were averaged within each Center to calculate a 
Center-level quality score. 
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Center-level scores were aggregated to calculate the average quality rating across 
sampled projects for the Center program as a whole, as well as for two subgroups of interest, the 
16 RCCs and the 5 CCs. Each Center was given equal weight in computing the overall program- 
wide rating for technical quality, as well as the mean ratings for the RCCs and the CCs 
respectively, for the sampled projects; within Centers, each project was given equal weight. 


Survey of Project Participants 

The purpose of the participant survey was to obtain client views of technical assistance 
from the Centers, particularly in the areas of relevance and usefulness. As the primary role of the 
RCCs is to provide technical assistance to the states in their regions, the clients for the RCCs 
included state-level staff. The clients for the CCs included both RCC staff and state-level staff, 
corresponding with the role of the CCs to build the capacity of the RCCs as well as to work with 
RCCs in providing technical assistance to states. 

The evaluation team developed two parallel survey forms for project participants to 
administer each year: one for state-level staff who participated in any Center project, and one for 
RCC staff who participated in a CC project (located in appendix G of this report). 40 In sampling 
participants to respond to surveys focused on specific projects, the goal was to identify a sizable 
number of participants, drawn from complete lists of all participants, so that their responses 
would collectively provide a picture of all participants’ views regarding the sampled projects. 

For each survey administration, the evaluation team drew samples of participants in the projects 
that were selected for expert panel review. In this way, expert panel ratings of quality and 
participant ratings of relevance and usefulness were gathered for the same set of projects. A brief 
description of the process used to calculate the measure of relevance and usefulness is provided 
at the end of this section. 

In each data collection year, Centers were asked to furnish full lists of all participants and 
their contact information for each sampled project. These lists made up the sample frame and 
included individuals who had participated in projects in numerous ways including: (1) serving on 
task forces, school support teams, and work groups associated with the project; (2) attending 
conferences, technical assistance retreats, and other meetings held as a part of the project; or (3) 
receiving written materials or other disseminated resources. State-level participants included 
staff who were employed by SEAs as well as employees of intermediate agencies, LEAs, 
schools, or other agencies who had responsibilities for state-level implementation of NCLB; they 
could be participants in both RCC and CC projects. RCC staff, as clients of the CCs, were 
participants solely in CC projects. 

After identifying the sampling frame, the team implemented a sampling strategy that 
combined random sampling from each project and a replacement strategy to minimize 


40 Since the survey asked respondents for their experiences with the Centers in relation to a specific project, the 
survey team provided each respondent with a list of the activities included in that project, based on the content of the 
project cover sheets provided by the Centers. For the on-line survey, this information was displayed on the first 
screen of the survey. For the paper version of the survey, the project-specific information was printed on yellow 
paper and inserted into the front of the booklet. 
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respondent burden. A power analysis was not conducted as part of the sampling plan for the 
participant survey. A sampling strategy was developed for selecting a sufficient number of 
participants to obtain fair representation of client views for each project, and across projects, 
while balancing respondent burden and data collection costs. The team drew a simple random 
sample of participants within each sampled project using the following sampling rules, based on 
the number of participants in the project: 

■ All participants in projects with 12 or fewer participants were sampled. 

■ A random sample of 12 participants was selected for projects with 13 to 25 
participants. 

■ A range of 12 to 48 participants were randomly selected to represent 48 percent of 
participants for projects with 26 to 100 participants. 

■ For each project with more than 100 participants, a random sample of 48 
participants was selected. 

Exhibit 2.10 provides an overview of the participant survey sampling and administration. 
The exhibit shows the overall process used in each year to select the sample in each of the data 
collection years. The number of respondents for each round of data collection was 1,208 project 
participants in 2006-07, 1,319 participants in 2007-08, and 1,035 participants in the final round 
in 2008-09. Details on the sampling frame for each administration and the number of participants 
at each stage of data collection are included in the appendix (see appendix B, exhibit B.4). 

The evaluation team initially administered the participant survey to the sampled 
respondents online. Nonrespondents were contacted by phone, sent reminder cards, and sent 
paper survey forms, all in an effort to obtain completed responses. Nonrespondents included 
participants who did not return a completed survey because they no longer worked for the state 
organization or RCC, as well as participants whose contact information was incorrect. 


Exhibit 2.10. 


Survey of project participants sampling and survey 
administration summary 


Sample frame — ► 


Study sample 


Survey 

administration 



EXHIBIT READS: The sample frame was made up of participants in RCC projects as well as those 
in CC projects. The resulting sample included state-level participants for RCC projects and a 
combination of RCC and state-level participants for CC projects. 
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Calculate Measures of Relevance and Usefulness 


For the relevance and usefulness questions in the participant survey, respondents were 
asked to rate each aspect of relevance and usefulness using a 5-point scale ranging from 5 (a very 
high degree) to 1 (a very low degree). 41 

For each of the data collection cycles, the evaluation team analyzed the properties of the 
items included in the survey for constructing indices of relevance and usefulness. Principal 
components analysis with no rotation was conducted on each set of items in the relevance and 
usefulness scales to determine the underlying dimensions represented by these items. For each 
scale, analyses of each year of data resulted in a one-factor solution comprised of the same items 
and with comparable levels of reliability. For the relevance scale, the Cronbach’s alpha was 0.93 
in 2006-07 and 0.94 in 2007-08 and 2008-09; it was 0.95 for the usefulness scale in all three 
years. Additional information on the factor loadings and psychometric properties for the three 
years of data collection is presented in appendix B (exhibits B.5 and B.6). 

In each year, the mean ratings at the respondent level were averaged so that each 
respondent for a given project contributed equally to a project-level rating. Thus, the relevance or 
usefulness rating at the project level was a mean of the ratings provided by sampled participants 
in that project (ranging in number from 1 to 48) who returned surveys. “ Next, the rating of each 
sampled project contributed equally to the computation of the mean rating across projects for 
each Center. Finally, the team calculated an overall mean across the 21 Centers and overall 
means for the 16 RCCs and the 5 CCs, respectively, for the relevance and usefulness ratings of 
the sampled projects. In calculating these overall means, each Center’s mean rating was 
weighted equally. 


Units of Analysis and Use of Statistical Tests in this Report 

In analyzing the quality, relevance, and usefulness data for this study for projects in the 
study sample, there were two important features of the data that had to be accommodated. First, 
the data were aggregated into several different units of analysis including the Center, project, and 
participant. Second, the process used to identify projects both for the sample frame and the study 
sample had an impact on the use of statistical tests in the resulting analyses. Both of these aspects 
of the data analysis are described below. 


41 The response category, “Not able to judge,” on the survey instruments indicated those respondents who were not 
appropriate for addressing the particular item and were, therefore, not included in analyses. In each of the data 
collection years, less than 1 percent of respondents indicated that they were “not able to judge” across either the 
eight items of the relevance scale or the 1 1 items of the usefulness scale. 

42 In each of the years across the eight items on the relevance scale, the rate of item nonresponse ranged from less 
than 1 percent to 2.5 percent. The rate of missing data among the 1 1 usefulness scale items was slightly higher with 
the percent of item-level nonresponse ranging from 0.1 to 2.8 percent. For both relevance and usefulness 
calculations, respondents with missing data on more than two of the items were excluded from the calculation, 
resulting in a small number of respondents being excluded from the analysis in each year. 


30 



Units of Analysis for Quality, Relevance, and Usefulness Ratings 


The quality, relevance, and usefulness ratings of the sampled projects were used in 
analyzing and reporting data for several different units of analysis: the projects sampled from the 
program as a whole (all 21 Centers), the projects sampled for the 16 RCCs and the 5 CCs 
respectively, subgroups of the projects, and (for relevance and usefulness) subgroups of the 
participants. For a Center’s mean rating on quality, relevance, or usefulness, the team weighted 
each sampled project from that Center equally; for a mean across the entire program or a set of 
Centers, the team weighted each Center’s mean equally. 

Several analyses were conducted for subgroups of projects across Centers — e.g., those 
projects that included a research synthesis among their products, or included training among their 
services. Each rater (an expert or survey respondent) had equal weight in the individual project’s 
rating, and each project had equal weight in the subgroup mean rating. 

Finally, some analyses were conducted for subgroups of participants across projects. For 
example, the ratings of relevance provided by participants who spent at least three days in a 
project activity were compared with the ratings provided by participants who spent less time. In 
these analyses, the unit was the participant, not the project, and the weighting was designed to 
permit generalization to all the participants across all the sampled projects. Thus the weight for 
each participant’s response was the inverse proportion of the number of respondents compared 
with the total number of participants in the population for that project. 

Number of Projects Varying with the Unit of Analysis . Readers will notice slight 
variations in the total number of projects shown in the exhibits and analyses throughout this 
report. These figures differ for two reasons. 

First, in each of the survey years, there was a small number of projects for which the 
evaluation team did not receive any completed surveys (two in 2006-07, four in 2007-08, and six 
in 2008-09). This meant that for any analysis incorporating participant data, the maximum 
number of projects in the analysis was reduced. For analyses that did not rely on participant data 
(e.g., the quality ratings), the team included all projects. 

Second, as described earlier in this chapter, in both 2006-07 and 2008-09 one project 
included in the sample was conducted jointly by an RCC and a CC. This “combined project” was 
one of the projects counted in analyses specific to RCC projects, and also one of the projects 
counted in analyses specific to CC projects. For analyses across all projects or all participants, 
however, the joint RCC/CC project or its participants were counted only once in the analysis. 


Use of Statistical Tests 

In this report, the presentation of results is sometimes framed by statistical tests, but in 
other instances those tests are not part of the presentation. The description below provides the 
rationale for these differences. 
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Analyses of Quality, Relevance, and Usefulness Ratings 


The findings presented in the last section of chapter 5 are based on the ratings of Center 
projects. For analysis by subgroups of project participants (last section of Chapter 5), inferential 
statistics and a level of statistical significance of p<.05 were used to decide which differences 
would be noted in the report. The evaluators adopted inferential statistics for analysis of the 
differences among project participants’ responses because the participants were selected using a 
stratified random sampling process. However, these results should not be generalized beyond the 
sample of projects selected for each year of the evaluation. 

For the interpretation of the analysis of project-level ratings for this study, the evaluation 
team adopted a difference of at least one-half of one standard deviation (calculated as the pooled 
standard deviation for all projects) in size as the minimum threshold for highlighting differences. 
Using Cohen (1988) as a conceptual framework, the evaluation team estimated Cohen’s d (an 
estimate of the effect size defined as the difference in means divided by the pooled standard 
deviation) and adopted the logic of Cohen for what would be considered a moderate difference. 

Although the report highlights general patterns in ratings from year to year, comparisons 
in ratings across year were not conducted. Because each study year’s sample of projects was 
selected through a non-probability procedure, evaluators determined that there was no basis for 
determining what portion of any differences in the distribution of project characteristics or 
ratings across years was the result of sampling bias and what portion reflected true changes in the 
operation of the projects. 


State Managers’ Assessment of Center Technical Assistance 

The data source for the state managers’ assessment of Center technical assistance in 
chapter 4 is the survey of state managers. The responses from this survey comprise a census of 
all state departments of education and not a probability sample. Thus, inferential statistics and 
assessments of statistical significance are not necessary, and any observed difference in the 
distribution of responses to the survey across time is considered a real change. While there may 
be measurement error, the estimate of whether any observed differences might be a result of 
chance incurred as part of a sampling process is not applicable. 
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3. How the Centers Operated 


An important context for assessing performance of the Centers is to understand Centers’ 
operations in each year and the changes in operations that occurred in the third and fourth 
program years (2007-08 and 2008-09). As described in chapter 1, ED established structures and 
expectations for the functioning of the Centers, including the division of responsibilities between 
the two types of Centers, the emphasis on applying scientifically based research and promising 
practices to build state capacity to carry out NCLB, and the flow of communication between and 
among the Regional Comprehensive Centers (RCCs), the Content Centers (CCs), and state 
agencies. 

This chapter describes the work of the Centers, addressing the following research 
questions: 


■ How did the Regional Comprehensive Centers and Content Centers operate as 

part of the Comprehensive Technical Assistance Center program? 

■ How did Centers develop, refine, and carry out their plans for technical 
assistance? How did they define their clients’ educational needs and 
priorities? 

■ What were the objectives of the technical assistance the Centers offered? 
What kinds of products and services were provided by the Centers? 

■ How did the Regional Comprehensive Centers and Content Centers 
coordinate their work? 

For this report a description of the Center operations focuses primarily on the most recent 
program year for which data were collected, the 2008-09 program year. To describe how Center 
operations have evolved over time, data from all three years are presented. 43 The chapter begins 
with a discussion of the processes by which Centers negotiated their work, describing ways in 
which they reportedly met the requirement to assess their client organizations’ needs. It provides 
information about the range of technical assistance objectives the Centers addressed and the 
products and services they delivered. Because the two types of Centers were expected to 
coordinate, it discusses ways in which they reportedly did so. The chapter concludes with a 
discussion of procedures for quality assurance that Centers reported. The analysis draws on 
information gathered from the Centers’ submitted Project Inventory Forms (PIFs), Centers’ 
annual management plans, cover sheets that Centers prepared for the expert reviews of their 


| ! In general, trend data include all three years of data collection with the following exceptions: management plans 
were reviewed for program years 2006-07 and 2008-09 only; more detailed reviews of the 2008-09 PIFs and 
accompanying cover sheets were performed to better understand the nature of Center work; questions addressing 
Center investments, processes, and working relationships with SEAs and other Centers were added in the last year of 
interview data collection and thus are only available in 2008-09. 
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sampled projects, and interviews with Center staff conducted by the evaluation team. 44 It also 
draws information about Center technical assistance from the survey of state managers 45 

Throughout the chapter, findings are presented separately for RCCs and CCs. As 
described in more detail in Chapter 1, RCCs were charged with providing direct assistance to 
states on an ongoing basis (“front-line assistance”) while CCs were charged with providing 
sound knowledge about their content areas to both states and RCCs. Because the two types of 
Centers differed in their charge and in their mix of clients, as explained in Chapter 1, one would 
expect differences in their procedures for planning and needs assessment, their objectives, and 
the products and services they delivered. 


Developing Plans: Identifying Client Needs and Priorities 

Before the start of each program year, the Centers were required to deliver a management 
plan to ED outlining the program of technical assistance they planned to provide to clients. A 
key expectation of the Centers was that they would organize their plans around the priorities and 
needs of client organizations. RCCs, charged with providing “front-line” assistance to a set of 
state clients, were expected to communicate with and serve those state agencies directly. The 
CCs served both RCCs and states as clients for their technical assistance. Because their technical 
assistance for states was to be provided in concert with RCCs, they could depend in part on the 
RCCs to alert them to state priorities. In addition to serving the RCCs and states, the CCs also 
served the needs of ED, responding to requests in specific content areas to advance federal 
priorities. The evaluation team gathered Center reports on how they identified client needs. 

According to the Centers, they gathered information to develop and refine their 
management plans for each year in a variety of formal and informal ways. A comparison of 
needs assessment techniques used for the 2006-07 program year with those used for the 2008-09 
program year shows the extent of continuity in these approaches to needs assessment. 

■ Centers conducted needs assessments annually with their respective primary 
client organizations. In 2008 the two most common modes that RCCs reported 
using to assess state needs were informal communications with state staff (all 16 
RCCs) and designation of a liaison to the SEA (15 of 16 RCCs) (exhibit 3.1). As 
compared with 2006, fewer RCCs reported communicating with chief state school 
officers in 2008 (11 in 2008 vs. 14 in 2006). For CCs, procedures used in 
assessing RCC needs in both 2006 and 2008 included communications with RCC 
staff (5 CCs in 2006 and 2008) and with RCC directors (4 of 5 CCs in 2006 and 
2008), and surveys (4 CCs in 2006 and 5 CCs in 2008). 


44 See chapter 2 for more information about the data sources and the procedures for gathering, coding, and analyzing 
the data reported in this chapter. 

45 A detailed description of the process used to gather and code the information discussed in this chapter can be 
found in appendix C and related supplemental tables are in appendix D. 
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Exhibit 3.1. 


RCC and CC interaction with clients in planning work, by year 



Number of Centers reporting this mode of interaction 
with clients 


RCCs 

CCs 

All Centers 


(N=16) 

(N=5) 

(N=21) 

Mode of interaction 

2006-07 2008-09 

2006-07 2008-09 

2006-07 2008-09 


Informal needs assessment 


through communication with 
state staff/ RCC staff 

15 

16 

5 

5 

20 

21 

Communication with chief state 
school officers/ RCC directors 

14 

11 

4 

4 

18 

15 

Surveys 

10 

8 

4 

5 

14 

13 

Designated liaison to SEA 

13 

15 

NA 

NA 

NA 

NA 

Liaison to SEA with office space 

or a u . . : i ^ — i : ^ ^ — . 

4 

3 

NA 

NA 

NA 

NA 


in SEA building 


NOTE: NA=Not Applicable. CCs were not asked about liaisons to SEAs because they worked nationally 
and were not expected to provide “front line” assistance. 

EXHIBIT READS: For the 2006-07 program year, 15 RCCs reported that they conducted informal needs 
assessment through communication with state staff; 5 CCs reported that they conducted informal needs 
assessment through communication with RCC staff. 

SOURCE: Center responses to binary interview questions. 


■ CCs assessed state-level needs both through direct communication with states 
and through communication with RCCs. For 2008-09, when an interview 
question asked about their means of assessing state needs, all 5 of the CCs 
reported interacting both with states and with RCCs in order to assess state needs 
(exhibit 3.2). Three of them volunteered a response that did not appear on the 
interview protocol: that they anticipated state-level issues and needs by following 
ED requests and national trends. 

■ CCs also reported continuing to respond to ED requests. In 2008, 4 of 5 CCs 
reported responding to requests made by ED; the same number reported doing so 
in 2006. 
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Exhibit 3.2. 


CC assessment of state needs, 2008-09 


Number of CCs 


Means of assessing state needs (N=5) 

Ongoing interaction with states 5 

Communication with RCCs about state needs 5 

Anticipation of issues by following ED initiatives 2 

and national trends 

Survey 1 


NOTE: The response, “anticipation of issues...” applies to three answers volunteered in response 
to a probe for “other” sources of input for planning. 

EXHIBIT READS: For the 2008-09 program year, all 5 CCs reported that they assessed the needs 
of the states through ongoing interaction. 

SOURCE: Center responses to binary interview questions. 


As a program providing services to the states, the Center program was expected to show 
responsiveness to needs and requests for technical assistance. However, Centers might not be in 
a position to respond to every client request. Whether in negotiating their management plans 
with clients or in discussing emerging needs over the course of the program year, Centers did not 
always agree to every request. Center staff provided the reasons for which they turned down 
client requests. 


■ In each year, more than half of the Centers reported that they had turned down 
a client request for assistance. The number of Centers that had turned down one 
or more requests increased within a narrow range over the years, from 12 of the 
21 in 2006-07 (9 of 16 RCCs and 3 of 5 CCs) to 13 of 21 in 2007-08 (9 of 16 
RCCs and 4 of 5 CCs), and 14 of 21 (10 of 16 RCCs and 4 of 5 CCs) in 2008-09. 

■ Among Centers that declined any client request for assistance in 2008-09, the 
substitution of a different type of assistance was reported by more than half of 
the RCCs but none of the CCs. Among the 10 RCCs that declined one or more 
requests, 7 RCCs worked out a different plan of assistance for the client; none of 
the CCs reported working out an alternate plan (exhibit 3.3). Of the 10 RCCs that 
had declined client requests, 5 RCCs reported that they could not honor a request 
because it fell outside their legitimate scope of work. Among the 4 CCs that 
declined requests, 2 CCs turned down requests that did not fit their priorities or 
their available staff time. 
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Exhibit 3.3. 


Reasons reported for declining technical assistance requests, 
2008-09 



RCCs 

(N=16) 

CCs 

(N=5) 

All 

(N=21) 

The Center did not turn down any requests for service 

6 

1 

7 

The Center turned down requests for service: 

10 

4 

14 

Turned down request but the Center and the client instead 
agreed on a plan for related, but different, technical assistance 

7 

0 

7 

Reasons for turning down request: 




A request fell outside the legitimate scope of work for a 
Center 

5 

1 

6 

A request was potentially legitimate Center work but did not 
fit this Center’s priorities for work with the state 

2 

2 

4 

Staff time and resources were already fully committed to 
other work 

2 

2 

4 

The Center did not have access to the needed expertise to 
carry out the request 

0 

1 

1 


EXHIBIT READS: For the 2008-09 program year, 6 of 16 RCCs did not turn down any requests for service. 
SOURCE: Center responses to standard response categories in interviews. 


Adjustments Made in Ongoing Work 

The Centers’ stated objectives could be found in their written annual management plans, 
but the ways in which they refined their plans might result in departures from these objectives. 
Annual management plans written in advance might in some ways be an asset to the Centers, 
enabling them to mount sustained efforts around their intended programs of work. At the same 
time, however, the plans might impose rigidity on Center technical assistance, impeding a 
flexible response to changing circumstances. The evaluation addressed this matter by assessing 
the extent to which Centers followed or adjusted their initial plans with respect to the topical 
coverage of the technical assistance, as stated in the objectives found in each annual management 
plan in the second and fourth program years (2006-07 and 2008-09). 

■ Across years, Centers continued to address a majority of the topics they had 

planned to address in each year, with some flexibility to make adaptations. The 
content of the Center management plans was compared with the projects Centers 
reported on the project inventory forms (PIFs) that they had conducted. In both 2006- 
07 and 2008-09 Centers delivered technical assistance on a particular topic in at least 
80 percent of the instances in which they stated that topic as part of their management 
plan objectives (80 percent in 2006-07, 84 percent in 2008-09) (exhibit 3.4 and 
appendix exhibit D.l). The analysis showed that of 117 topic-related objectives in the 
2008-09 plans, the Centers provided services and products on 98 of them. The fact 
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that some planned cases 46 (117-98=19) were not listed on the PIFs indicates that 
16 percent of the cases (19 of the planned 117) were not conducted. The 
corresponding figure was 20 percent for 2006-07. Looking at the work actually 
conducted, in 36 percent of cases ([ 152-98J/152) the Center provided technical 
assistance on a topic not initially cited in its planned objectives in 2008-09. The 
corresponding figure was 38 percent for 2006-07. 

■ There was continuity across years in the topics most often added to or deleted 
from Center agendas during program years; the topics of English language 
learners and response to intervention were added in more instances than other 
topics, while special education curriculum, instruction, and professional 
development was deleted in more instances than other topics . In Exhibit 3.4, 
comparing columns 3 and 2 provides an indication of topics not included in a 
Center’s management plan but in which technical assistance was delivered by that 
Center (additions). Comparison of columns 1 and 2 indicates topic areas in which 
Centers planned but did not deliver a project (deletions). Eight Centers conducted 
work in 2008-09 related to English language learners but had not included that 
topic among their objectives for the year. The other topics added by the next- 
largest number of Centers were response to intervention (7 Centers) and migrant 
education (6 Centers). For 2006-07 the additions found in the largest number of 
Centers were response to intervention (7 Centers), English language learners 

(5 Centers), highly qualified teacher provisions (5 Centers), and supplemental 
educational services (5 Centers). There were 12 of the 22 topics on which Centers 
initially stated an objective but did not deliver a project in 2008-09, and 10 such 
topics in 2006-07. The topic for which work was deleted by the largest number of 
Centers in both years was special education curriculum, instruction, and 
professional development (4 Centers in 2008-09, 5 in 2006-07). 

■ Adaptations in the topical focus of plans were more common among RCCs than 
CCs. For 2008-09, the RCCs added work on 21 of 22 topics while CCs did so on 
4 of 22 topics (exhibit 3.5). The same difference was found in 2006-07, when the 
figures were 19 of 22 topics for RCCs and 7 of 22 for CCs (appendix exhibit D.2). 
Deletions of planned work also spanned more topics among RCCs than CCs 
across years, being observed for 12 of 22 topics among RCCs in 2008-09 (11 of 
22 in 2006-07) and 3 of 22 topics among CCs in 2008-09 (2 of 22 in 2006-07). 


46 Using each Center's management plan, the evaluation team coded for each Center whether it had specifically stated an 
intention/objective to address work in each of the 22 topics (i.e., the first column of numbers in exhibit 3.4 indicates the number 
of Centers that stated an objective of conducting any work in each topic, beginning with the topic of state systems of support). 
Using the PIFs, the evaluation team coded project topics to determine the number of Centers that actually did work in each topic 
(i.e., the last column of numbers, by topic, in exhibit 3.4). 
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Exhibit 3.4. Topics on which Centers stated objectives and/or delivered 
projects, 2008-09 


Number of Centers (N=21) 


Topic 

Objective on 
topic stated in 
management 
plan 

(D 

Objective on topic stated 
in management plan AND 
at least one project on 
topic reported on PIF 
(2) 

At least one 
project on 
topic reported 
on PIF 

(3) 

Total cases of a Center setting an 
objective and/or reporting a project 

117 

98 

152 

Components of effective systems of 
support-state, district, school 

16 

16 

17 

English language learners 

9 

8 

16 

Highly qualified teacher provisions of NCLB 

7 

7 

9 

Teacher preparation and induction 

7 

5 

7 

Teacher professional development 

4 

4 

6 

Data use /data-driven decision making 

7 

5 

5 

Assessment design 

7 

5 

6 

Formative assessment 

5 

4 

5 

Special education-curriculum, instruction, 
and professional development 

5 

1 

2 

Response to Intervention (Rtl) 

5 

4 

11 

Special education-assessment 

4 

4 

6 

High school redesign/reform 

8 

7 

9 

Transition to high school 

1 

1 

3 

Dropout prevention 

3 

3 

5 

Mathematics 

7 

7 

9 

Adolescent literacy 

5 

5 

8 

Reading 

2 

2 

3 

Supplemental educational services (SES) 

4 

2 

4 

Parent involvement 

5 

4 

7 

Migrant education 

2 

1 

7 

Data management compliance 

3 

3 

5 

Indian/Native American education 

1 

0 

2 


EXHIBIT READS: For the 2008-09 program year, for the topic “Components of effective systems of support — state, 
district, school,” 16 Centers had a related objective in their respective annual management plans; of these, all 16 
reported projects on the topic in their PIF; and a total of 17 Centers had projects on the topic, whether or not they had 
stated an objective related to it in their management plan. 

SOURCES: Center management plans for 2008-09 and PIFs prepared by Centers in consultation with evaluation 
team. 


39 



Exhibit 3.5. 


Topics on which RCCs and CCs stated objectives and/or 
delivered projects, 2008-09 



Number of RCCs (N= 

16) 

Number of CCs (N=5) 



Objective 

Objective on 


Objective 

Objective on 



on topic in 

topic in plan 

Project(s) 

on topic in 

topic in plan 

Project(s) 


manage- 

AND project(s) 

on topic 

manage- 

AND project(s) 

on topic 


ment plan 

on topic on PIF 

on PIF 

ment plan 

on topic on PIF 

on PIF 


(D 

(2) 

(3) 

(4) 

(5) 

(6) 

Total cases 

98 

82 

130 

19 

16 

22 

Components of effective 
systems of support-state, 
district, school 

15 

15 

16 

1 

1 

1 

English language 
learners 

7 

6 

12 

2 

2 

4 

Highly qualified teacher 
provisions of NCLB 

6 

6 

8 

1 

1 

1 

Teacher professional 


R 

7 

1 

0 

0 

development 

u 

J 


Teacher preparation and 

Q 

Q 


1 

1 

1 

induction 

o 

O 


Data use / data-driven 


A 

A 

1 

1 

1 

decision making 

b 

4 

4 

Assessment design 

6 

5 

6 

1 

0 

0 

Formative assessment 

4 

3 

4 

1 

1 

1 

Special education- 
curriculum, instruction 

A 

1 

O 

1 

0 

0 

and professional 
development 

4 

<L 





Response to Intervention 
(Rtl) 

5 

4 

9 

0 

0 

2 

Special education- 
assessment 

2 

2 

4 

2 

2 

2 

High school 

7 


8 

1 

1 

1 

redesign/reform 



Transition to high school 

0 

0 

2 

1 

1 

1 

Dropout prevention 

2 

2 

4 

1 

1 

1 

Mathematics 

6 

6 

8 

1 

1 

1 

Adolescent literacy 

4 

4 

6 

1 

1 

2 

Reading 

Supplemental 

1 

1 

2 

1 

1 

1 

educational services 
(SES) 

3 

1 

3 

1 

1 

1 

Parent involvement 

5 

4 

6 

0 

0 

1 

Migrant education 

2 

1 

7 

0 

0 

0 

Data management 

3 

3 

5 

0 

0 

0 

compliance 




Indian/Native American 
education 

1 

0 

2 

0 

0 

0 


EXHIBIT READS: For the 2008-09 program year, tor the topic “Components of Effective Systems of Support — State, District, 
School,” 15 RCCs reported a related objective in their management plans; of these, all 15 reported projects on the topic; and 
a total of 16 RCCs had projects on the topic, whether or not they had originally stated an objective related to it. 


SOURCES: Center management plans for 2008-09 and PIFs prepared by Centers in consultation with evaluation team. 
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The Nature of Center Work 


Centers carried out a variety of technical assistance activities across and within projects. 
Their work reflected change and continuity across years as well as decisions about priorities 
negotiated with clients for each year. We discuss here the types of project investments reported 
by Centers in 2008-09, the topics addressed across all projects in all three years, the activities 
carried out in sampled projects in all three years, and the nature of the sampled projects in 2008- 
09 on a high-priority topic, that of statewide systems of support. 

Throughout this section we refer to seven distinct types of activities 47 , which the team 
identified and defined as follows: (1) ongoing consultation and follow-up through multiple 
service contacts over time, either to fulfill repeated requests or to follow up with individuals who 
participated in another assistance activity; (2) research collections and syntheses distributed by a 
Center, whether developed by that Center or elsewhere; (3) engagement of participants in project 
planning, defined as going beyond needs assessment to include opportunities for at least some 
participants to shape the specific assistance; (4) training events, designed to impart skills and 
equip participants to carry out a particular program or strategy; (5) taskforce meetings and work 
in which a Center supported a group that was itself constituted as a task force to address a state 
purpose; (6) conferences, defined as single events in which multiple speakers or discussants 
presented information; and (7) support for development of a formed plan to implement a program 
or policy, where the Center supported work by participants in a state. 


Center Investments 

For the fourth program year, 2008-09, interviewers asked the Centers to identify their 
investment priorities with respect to project duration and change, as well as to the types of 
technical assistance activities that were priorities for their clients and themselves. Another 
perspective on these matters was provided by the state managers who were asked in surveys 
about the types of technical assistance that their state received from Centers. 

■ Center directors reported their largest investments to be in multi-year projects 
following “ a long term plan.” This response was given by 13 of the 16 RCCs and 
4 of the 5 CCs (exhibit 3.6). For three RCCs, the greatest investment was in multi- 
year projects that had undergone “substantial shifts from the originally planned 
participants, activities, or purposes.” One CC reported the greatest investment in 
projects that were self-contained within 2008-09. 


47 At the project level, the cover sheets that Centers submitted along with their materials for expert review provided 
data on the types of activities and resources that the project offered to some or all of its participants. See Chapter 2 
for additional details. 
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Exhibit 3.6. Reported investment of Center resources in long-term vs. short- 
term projects, 2008-09 


Number of Centers reporting as the 
largest investment of resources 
for 2008-09 

Types of Center projects 

RCCs 

(N=16) 

CCs 

(N=5) 

All 

(N=21) 

Multi-year project(s) that followed a long-term plan 
for participants, activities, and purposes 

13 

4 

17 

Multi-year project(s) with substantial shifts from the 
originally planned participants, activities, or 
purposes 

3 

0 

3 

Projects that were self-contained within a single 
year rather than extending across years 

0 

1 

0 


EXHIBIT READS: For the 2008-09 program year, 13 RCCs reported that their largest investment of resources 
was in multi-year projects that followed a long-term plan for participants, activities, and purposes. 

SOURCE: Center responses to standard response categories during interviews. 


When asked in interviews about the changes in Center work from 2006 to 2009, Center 
directors from 6 RCCs and 3 CCs described Center assistance as becoming more long term and 
ongoing, whereas it previously was more likely to focus on delivering discrete activities or 
materials. Among the 16 RCCs, 9 reported their work as increasingly focused on systemic and 
structural issues at the state level, with more efforts to build sustainable structures and processes. 
In addition to a shift in focus, 9 RCCs also described providing more “on-call assistance” to SEA 
leaders to think through and help solve pressing problems or plan for improvement. Four of the 5 
CC directors reported an increase in direct assistance to state or RCC clients, such as consultative 
assistance in applying research. 

Centers could address state needs and priorities in a wide range of ways, and in making 
choices among activities they had to strike balances. In negotiating their management plans and 
project activities, they had to recognize clients’ demand for particular modes of technical 
assistance while using their professional judgment about what would be most productive in 
meeting Center objectives. 

■ Ongoing consultation was the type of project activity cited by the largest 
numbers of RCCs and CCs as (1) most requested by clients, (2) the largest 
investment, and (3) the most important for achieving Center aims in 2008-09. 
Asked to select from a general list of seven types of project activities and 
resources, the directors of 5 of the 16 RCCs and 2 of the 5 CCs said they made the 
greatest investment of resources in “ongoing consultation” in 2008-09 (exhibit 
3.7). Directors also identified this activity as the one most requested by clients (4 
of the 16 RCCs and 4 of the 5 CCs), and the one most important for achieving 
Center objectives (6 of the 16 RCCs and 3 of the 5 CCs). 
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Exhibit 3.7. 


RCC and CC activities and resources by reported client 
demand, investment, and importance to objectives, 2008-09 



Most requested by 
clients 

Largest investment 

Most important to 
achieve objectives 

Activities and 
resources 

RCCs 

CCs 

All 

Centers 

RCCs 

CCs 

All 

Centers 

RCCs 

CCs 

All 

Centers 

Ongoing consultation 
and follow-up 

4 

4 

8 

5 

2 

7 

6 

3 

9 

Research collections 
and syntheses 

1 

1 

2 

0 

1 

1 

0 

0 

0 

Engagement of 
participants in project 
planning 

2 

0 

2 

3 

0 

3 

4 

0 

4 

Training events 

4 

0 

4 

3 

0 

3 

1 

0 

1 

Task force meetings 
and work 

3 

0 

3 

3 

0 

3 

3 

0 

3 

Conferences 

0 

0 

0 

0 

1 

1 

0 

1 

1 

Support development 
of a formal plan to 
implement a program 
or policy 

2 

0 

2 

2 

1 

3 

2 

1 

3 


EXHIBIT READS: For the 2008-09 program year, four of the 1 6 RCCs indicated in interviews that ongoing 
consultation and follow-up was the service most requested by their clients. 


SOURCE: Project cover sheets prepared by Centers for the expert review of project materials and coded by the 
evaluation team. 


■ State managers’ survey responses across the three years corroborated the 

Centers’ reports of continuity in the work, and indicated that Centers provided a 
combination of hands-on help and delivery of knowledge resources. The largest 
number of state managers, weighted, said in each year that the Centers had 
provided “major” assistance with facilitating work groups, with the percentage 
ranging from 43 to 65 percent across years) (exhibit 3.8). The next most often 
reported type of assistance was hands-on help with designing, delivering, or 
convening professional development for local educators (ranging from 29 to 55 
percent), followed by delivery of information about policies and practices in other 
states (ranging from 28 to 46 percent) or about research findings (ranging from 25 
to 45 percent). 


48 For 20 or more states in each year, the evaluation team received completed responses from more than one state 
manager. As the state was the primary unit of analysis in analyzing data from the state manager survey for this 
report, the state managers’ responses were weighted to ensure that each state was equally represented in all summary 
statistics while taking into account the variation in responses within each state. The weighting procedure, where 
each response was weighted by the inverse of the number of managers responding from that state, ensured that each 
state was equally represented when the team aggregated responses across states to describe the distribution of 
responses. See chapter 2 for a detailed description of the sampling and analysis procedures. 
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Exhibit 3.8. 


State managers’ reports of assistance from Centers, by type of 
assistance, by year 


Percent of state managers 
(weighted) reporting “major” 
assistance from any Center 

Type of assistance 

2006-07 

2007-08 

2008-09 

Facilitating work groups or committees 

43% 

56% 

65% 

Designing, delivering, or convening PD and 
conferences for local educators 

29 

45 

55 

Collecting/disseminating information about 
policies and practices in other states 

28 

33 

46 

Synthesizing and disseminating research 
findings 

25 

44 

45 

Analyzing data or conducting needs 
assessments 

16 

35 

34 

Developing tools for monitoring programs 

15 

35 

32 

Reviewing state plans and policies 

21 

31 

29 

Assisting with a response to federal planning 
and reporting requirements 

15 

20 

21 

Completing routine tasks more efficiently 

13 

12 

17 


EXHIBIT READS: For the 2006-07 program year, 43 percent of state managers (weighted) reported that 
their state had received “major” assistance from their RCC or any CC that consisted of facilitating work 
groups or committees. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in 
instances where more than one manager from a state responded. 


Topics of Projects 

As discussed earlier in this chapter, the Centers classified all of their projects for each 
year of the study according to the primary topics addressed, using a list of topics developed for 
this evaluation, to complete their annual project inventory forms (PIFs). The number and percent 
of all projects addressing each topic gives an indication of the substantive priorities identified by 
Centers and their clients in each year. Across 23 topics (including “other”), one was the focus for 
the largest share of projects in each year: 

■ A prioritized topical focus for Center projects in each year was assistance with 

“ systems of support,” addressing support for improvement in struggling schools 
and districts. Among all projects, 19 percent addressed this topic in 2006-07, 

25 percent in 2007-08, and 21 percent in 2008-09 (exhibit 3.9). After “systems of 
support”, the topic area with the second largest number of projects each year 
focused on English language learners (7 percent of all projects in both 2006-07 
and 2007-08, and 10 percent in 2008-09). Center services were otherwise widely 
dispersed across topic areas, and no other topic was the focus of as many as 10 
percent of projects in any year. 
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Exhibit 3.9. Percent of projects on PIFs, by topic and by year 



2006-07 

2007-08 

2008-09 

Total number of projects 

365 

346 

332 

Topics 


Percent of total 


Components of effective systems of 
support — state, district, school 

19% 

25% 

21% 

English language learners 

7 

7 

10 

Highly qualified teacher provisions of 
NCLB 

4 

6 

5 

Teacher professional development 

4 

2 

2 

Teacher preparation and induction 

2 

2 

2 

Data use / data-driven decision making 

2 

1 

2 

Assessment design 

1 

2 

2 

Formative assessment 

3 

3 

2 

Special education-curriculum, 
instruction and professional 
development 

2 

2 

1 

Response to Intervention (Rtl) 

3 

6 

6 

Special education-assessment 

1 

1 

3 

High school redesign/reform 

4 

3 

5 

Transition to high school 

2 

1 

1 

Dropout prevention 

1 

1 

2 

Mathematics 

5 

6 

8 

Adolescent literacy 

5 

5 

4 

Reading 

7 

3 

2 

Supplemental educational services 
(SES) 

5 

3 

2 

Parent involvement 

3 

3 

3 

Migrant education 

3 

4 

3 

Data management compliance 

4 

3 

3 

Indian/Native American education 

1 

2 

2 

Other 

13 

7 

9 


EXHIBIT READS: For the topic “Components of effective systems of support — state, district, school,” there 
were 71 projects on all PIFs for the 2006-07 program year, which was 19 percent of the total number of 
projects for that year. 

SOURCE: PIFs prepared by Centers in consultation with evaluation team. 
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Types of Activities and Resources 


A closer look at the Center projects in each year’s sample provided further insight into 
their scope and the activities they encompassed. For each year, the team’s analysis of sampled 
projects (through procedures described in chapter 2) provides information about the nature of 
Center technical assistance activities or resources. Although the sample of projects is not 
statistically representative of the Centers’ work, project sample selection for review of each 
Center’s work favored the most dominant projects and included more than half (56 percent) of 
the designated major or moderate projects. 

Here we focus particularly on comparing activities across RCCs’ and CCs’ sampled 
projects across years. We first remind readers that a project is a unit of analysis that may 
encompass many different activities: 

■ Projects entailed multiple activities, with RCC projects averaging between 3 and 
4 activities in each year and CC projects averaging between 2 and 3 activities. 
The mean number of activities in RCC projects was 3.2 in 2006-07, and 3.9 in 
2007-08 and 2008-09. In CC projects, the mean was 2.4 in 2006-07, 2.5. in 2007- 
OS, and 2.8 in 2008-09. 

Exhibit 3.10. Sampled RCC and CC projects by types of activities and 
resources, by year 



Percent of sampled RCC 
projects 

Percent of sampled CC 
projects 

Activities and resources 

2006-07 

(n=96) 

2007-08 

(n=96) 

2008-09 

(n=93) 

2006-07 

(o=27) 

2007-08 

(n=26) 

2008-09 

(n=26) 

Ongoing consultation and follow- 
up 

82% 

93% 

91% 

22% 

15% 

62% 

Research collections and 
syntheses 

54 

53 

53 

74 

85 

77 

Engagement of participants in 
project planning 

45 

63 

59 

30 

31 

27 

Training events 

43 

55 

59 

37 

50 

50 

Task force meetings and work 

50 

58 

56 

7 

8 

8 

Conferences 

27 

35 

40 

63 

42 

38 

Support development of a formal 
plan to implement a program or 
policy 

19 

32 

31 

7 

15 

23 


EXHIBIT READS: For the 2006-07program year, 82 percent of sampled RCC projects included ongoing consultation 
and follow-up. 


SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. In addition to serving 
as resource material for the expert reviewers, these cover sheets were coded by the evaluation team. 
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■ Across years, the kinds of support provided by the RCCs and CCs differed in 
ways that were consistent with the model of technical assistance envisioned by 
ED. The guidance given by ED through the Center grant competition and 
afterwards laid out a particular structure for the Centers’ work: RCCs would 
specialize in interactions with state clients to provide frontline assistance while 
CCs would specialize in activities that required a content focus. In each year, the 
most common activity found in sampled RCC projects was “ongoing consultation 
and follow-up” (82 percent in 2006-07; 93 percent in 2007-08; and 91 percent in 
2008-09) (exhibit 3.10). In CC sampled projects, it was “research collections and 
synthesis” (74 percent in 2006-07, 85 percent in 2007-08, and 77 percent in 2008- 
09), an activity that fewer RCC projects included (53 or 54 percent in each year). 

■ While the two types of Centers each retained a focus on activities distinctly 
associated with the original program design, their ways of working became more 
similar over the years. For example, in a departure from past CC practice, more 
than half of the sampled CC projects in 2008-09 (62 percent) included ongoing 
consultation and follow-up, an activity found in the great majority of RCC projects 
(at least 82 percent in every year). Conferences, which had been more than twice as 
common in the CC project sample as in the RCC project sample in 2006-07 

(63 percent of projects vs. 27 percent) were found in 38 percent of the CC projects 
and 40 percent of the RCC projects in 2008-09 (exhibit 3.10). 
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Projects on “Effective Systems of Support - State, District, School” 

In every year, the topic addressed by the largest number of Center projects was “components of 
effective systems of support — state, district, school.’’ This included technical assistance with the statewide 
systems of support (SSOS) that SEAs are required to mobilize to help struggling schools and districts 
improve student performance. An SSOS comprises an infrastructure for the delivery of onsite help, and 
strategies and materials designed to guide local improvement. For the sampled projects addressing 
statewide systems of support in 2008-09, further detail is provided about patterns of activities conducted 
and resource materials used. Among the 33 sampled projects on systems of support in 2008-09, Centers 
assisted in one or both of two ways: 

■ Working with SEA decision makers in redesigning their state’s SSOS infrastructure, 
revising strategies for local improvement, or both (24 projects). SSOS redesign efforts 
included reorganization of the SEA and delineation of responsibilities in the SSOS. 
Revising strategies centered on the content to be delivered, for example, approaches to 
instructional coaching or a framework of indicators to guide school improvement. 

■ Teaching (or helping teach) support providers in carrying out their work in the field (15 
projects). The content of project training centered on using the SEA improvement and 
planning tools with schools and districts (e.g., planning templates, guides, rubrics, self- 
assessments). In 12 of these projects, Centers also assisted in developing the state 
materials for use in the field. 

Among the materials that RCCs or CCs employed in their work with SEAs or support providers in 
these projects, the three used most often were published by the Content Center on Innovation and 
Improvement (CM) in 2007: 

■ “Handbook on Statewide Systems of Support’’ (9 projects). This 294-page handbook 
reviews research related to SSOS and principles for improving schools. It includes 
chapters on the state role, a conceptual framework for an effective SSOS, case studies 
of RCC support, and tools for assessing SSOS. 

■ “Strengthening the Statewide System of Support: A Manual for the Comprehensive 
Center and the State Education Agency” {7 projects). This 99-page manual is designed 
to guide an RCC in facilitating SSOS self-assessment with an SEA. 

■ “Handbook on Restructuring and Substantial School Improvement” (6 projects). This 1 25- 
page handbook offers modules with action principles on district frameworks, school board 
and central office roles, school change, learning-focused leadership, monitoring and 
changing instruction, and systems for improved teaching. It also provides approximately 
150 indicators of success in school restructuring. This manual builds on the publication 
School Restructuring Under NCLB: What Works When?, produced by Public Impact and 
the Center for Comprehensive School Reform and Improvement. 


Working Relationships in Technical Assistance Delivery 

The delivery of technical assistance could be affected by several types of working 
relationships: the coordination of work between RCCs and CCs in the two-tier system; the 
Centers’ working relationships with their clients; and the engagement of knowledgeable 
individuals in quality assurance. Based on the design of the center system, the RCCs and CCs 
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were expected to work together to serve state clients, and at the same time CCs were to provide 
technical assistance to the RCCs. Impediments could arise in RCC-CC coordination, in the 
provision of technical assistance to states, and in CCs’ assistance to RCCs. Also, in the process 
of preparing technical assistance activities and resources for delivery to clients, Centers drew on 
various sources of quality assurance. These various aspects of coordination and service delivery 
were examined in the evaluation based on the sampled projects across the three years (2006-07, 
2007-08, and 2008-09). Questions addressing Center investments, processes, and working 
relationships with SEAs and other Centers were added in the last year of data interview 
collection, and thus are only available in 2008-09. 


RCC-CC Coordination 

Charged with working together to assist the states, RCCs and CCs had to communicate 
and coordinate. Cooperation across Centers could be manifested not only in communication 
across organizations but also within specific projects: technical assistance projects of each type 
of Center could potentially incorporate help from the other type of Center. The evaluation 
tracked each of these types of coordination. 

For a description of coordination between CCs and RCCs, each CC director was asked 
about types of activities it had carried out in 2008-09 with at least some of the RCCs. 
Corroboration was sought from RCCs, whose directors were asked whether each of the CCs (by 
name) had coordinated in these ways with them. 

■ All RCCs and CCs reported communicating at least monthly with at least one 
Center of the other type in 2008-09, but working relationships varied across the 
specific Centers. When asked about their coordination practices, all 16 RCCs 
reported receiving knowledge resources from CCs and communicating at least 
monthly with CCs (exhibit 3.11); all 5 of the CCs also reported these coordination 
practices. However, there was variation in the number of RCCs reporting that 
individual CCs had coordinated with them. For example, 15 RCCs reported 
monthly communication with “CC A,” whereas 7 RCCs reported monthly 
communication with “CC D.” 

■ All but one of the RCCs reported teaming up with a CC to provide assistance to 
states, but this type of coordinated assistance occurred more often with 
particular CCs. When asked about their coordination in providing assistance to 
states, 15 RCCs reported teaming up with CCs and all 5 CCs reported teaming up 
with RCCs. However, the extent to which this type of coordination occurred 
varied by CC. For example, 14 RCCs reported working with 1 CC while 7 RCCs 
reported working with another CC. 

■ Communication about promising practices in states was the least common type 
of coordination. Two of the 5 CCs reported soliciting information about 
promising state practices from RCCs. Ten of the 16 RCCs reported offering such 
information to CCs (exhibit 3.11), with 2 to 7 RCCs reportedly communicating 
with each specific CC about promising state practices. 
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Exhibit 3.11. RCC-CC coordination reported by Center directors, 2008-09 



Number of 

Number of RCCs reporting this 

Number of 


RCCs 

type of coordination with 

CCs 


reporting this 


individual CCs 


reporting 


type of 



(N=16) 



this type of 


coordination 






coordination 


with one or 






with one or 


more CCs 

CC 

CC 

CC 

CC 

CC 

more RCCs 

Type of coordination 

(N=16) 

A 

B 

c 

D 

E 

(N=5) 

RCC received CC knowledge 
resources (materials or 
experts); CC provided 
knowledge resources 

16 

16 

15 

13 

15 

14 

5 

RCC and CC communicated 
at least monthly 

16 

15 

14 

10 

7 

11 

5 

RCC and CC teamed up to 
provide technical assistance 
to states 

15 

14 

7 

8 

12 

9 

5 

RCC provided information 
about promising state 
practices to CC(s); CC 

10 

7 

r 

R 


R 

p 

solicited information about 

/ 

O 

u 

2- 

o 

£_ 

promising state practices 
from RCCs 









EXHIBIT READS: For the 2008-09 program year, all 16 RCCs reported that they received knowledge 
resources from one or more of the CCs. Sixteen RCCs also reported this type of coordination with CC “A.” 

SOURCE: Center responses to standard response categories during interviews. 


For insight into the nature of joint RCC-CC work at the project level, the study team 
determined whether each RCC project sampled for the evaluation in each year incorporated 
specific types of input from one or more CCs, and vice versa. 

■ In all three years, more than half of Center projects were completed by a single 
type of Center, although the types of roles that Centers played when involved in 
each others’ projects were consistent with the design of the Center system. The 
CCs, which were expected to focus on the research in a content area, provided 
substantive materials, assistance, or advice in connection with between 32 percent 
and 48 percent of sampled RCC projects across years (exhibit 3.12). The RCCs, 
which were expected to maintain contact with the states in their regions, recruited 
participants or brokered services for 35 to 38 percent of sampled CC projects 
(exhibit 3.13). 

■ Across years, there were upward trends in CCs and RCCs providing direct 
technical assistance in the other type of Center’s projects. Among sampled RCC 
projects, the percentage that included direct assistance from CC staff was 

18 percent in 2006-07, 22 percent in 2007-08, and 30 percent in 2008-09 (exhibit 
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3.12). The corresponding figures for RCC delivery of technical assistance in 
sampled CC projects were 11 percent, 12 percent, and 38 percent (exhibit 3.13). 


Exhibit 3.12. Sampled RCC projects by CC contribution, by year 



Percent of sampled RCC 
projects 


2006-07 

(n=96) 

2007-08 

(n=96) 

2008-09 

(n=93) 

RCC project had no CC contribution 

52% 

67% 

52% 

RCC project had a CC contribution 

48 

32 

47 

CC provided materials used in this RCC project 

44 

28 

35 

CC delivered technical assistance to project 
participants 

18 

22 

30 

CC provided advice to the RCC on this project 

17 

17 

26 


EXHIBIT READS: For the 2006-07 program year, there was no CC contribution in 52 percent of sampled 
RCC projects. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials and coded 
by the evaluation team. 


Exhibit 3.13. Sampled CC projects by RCC contribution, by year 



Percent of sampled CC 
projects 


2006-07 

(n=27) 

2007-08 

(n=26) 

2008-09 

(n=26) 

CC project had no RCC contribution 

63% 

62% 

58% 

CC project had an RCC contribution 

37 

38 

42 

RCC recruited participants or brokered service 

37 

38 

35 

RCC delivered technical assistance to project 
participants 

11 

12 

38 


EXHIBIT READS: For the 2006-07 program year, there was no CC contribution in 63 percent of sampled 
CC projects. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials and coded 
by the evaluation team. 
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Center Reports of Barriers to Achieving Technical Assistance Aims 

The effective delivery of technical assistance depended on the Centers’ working 
effectively with their clients. Centers had the opportunity to describe in interviews the 
circumstances that might have impeded these working relationships. Both RCCs and CCs 
described the barriers that they perceived in achieving their objectives for assisting states. (In 
chapter 4, Centers’ perceptions are compared with those reported by state managers.) Each type 
of Center, from its own perspective, also addressed barriers impeding the delivery of CC 
technical assistance to RCCs as clients. 

■ Turnover in staff within state offices or intermediary units was the barrier most 
widely reported by both types of Centers, but reported by RCCs to be the greatest 
barrier. Staffing turnover was reported as a barrier by 10 of the 16 RCCs and 3 of 
the 5 CCs (exhibit 3.14). Six RCCs (but no CCs) identified it as the greatest 
barrier. Turnover at the leadership level was a reported barrier for 8 RCCs and 3 
CCs, and the greatest barrier for 2 RCCs. 

■ The two greatest barriers CCs reported were policy shifts at the state level and a 
mismatch between state priorities and their own scopes of work. Two of the 

5 CCs reported policy or priority shifts as a barrier, and both of them termed this 
the greatest barrier to achieving their objectives with states (exhibit 3.14). Three 
of the 5 CCs reported that “a state’s most important priorities for assistance fell 
outside the Center’s scope of work,” and 2 of them viewed this as their greatest 
barrier. 

■ Four RCCs raised an issue that had not been on the list of responses: cuts in 
state staffing and budgets. In response to the prompt, “other,” 4 of the 16 RCCs 
brought up the states’ fiscal difficulties as an impediment to technical assistance, 
noting that state layoffs or hiring freezes were challenges for their assistance plans 
(exhibit 3. 14). 49 


49 Since the Centers were not specifically prompted with this response option, we cannot say how many others 
would have reported it as a barrier if asked. (An additional 4 RCCs and 1 CC gave a range of different responses to 
the “other” prompt, as shown in the exhibit.) 
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Exhibit 3.14. Centers’ perception of barriers to achieving objectives in 
delivering technical assistance to states, 2008-09 


Number of Centers reporting: 



Barrier 

Greatest barrier 

Barrier to achieving technical assistance 
objectives 

RCC 

(N=16) 

cc 

(N=5) 

RCC 

(N=16) 

CC 

(N=5) 

No barriers reported to achieving objectives in 
serving states 

1 

1 

1 

1 

A state office, division, or intermediary unit 
experienced turnover in staffing 

10 

3 


0 

A state experienced turnover in leadership 

8 

3 


0 

A policy or priority shift at the state level caused the 
Center’s assistance to be less helpful than it might 

8 

2 

1 

2 

There was a lack of coordination or communication 
within a state agency 

5 

1 

1 

0 

State staff did not have time to work with the Center 

5 

0 

1 

0 

Cuts in state staff and budgets* 

4 

0 

1 

0 

A state would have preferred to locate and contract 
directly with experts or consultants rather than 
working with the Centers 

3 

0 

1 

0 

A state’s most important priorities for assistance fell 
outside the Center’s scope of work 

2 

3 

0 

2 

A state secured most of the technical assistance it 
needed from other sources 

2 

1 

0 

0 

The process of negotiating a work scope and 
organizing projects took too long 

1 

0 

0 

0 

Center staff were not able to spend as much time 
working with a state as the state would have liked 

1 

0 

0 

0 

State clients placed a higher priority on completing 
short-term work than on addressing long-term 
purposes 

1 

0 

0 

0 

The Center was unable to develop a productive 
working relationship with a state 

0 

1 

0 

0 

The Center did not have the expertise a state 
needed 

0 

0 

0 

0 

Other 

4 

1 

2 

0 


NOTE: This response was volunteered by multiple respondents in response to the “other” prompt and was 
re-coded as a separate response. 


EXHIBIT READS: For the 2008-09 program year, One RCC reported no barriers to achieving objectives in 
serving states. Ten RCCs reported that a barrier to achieving its technical assistance objectives was that a 
state office, division, or intermediary unit experienced a turnover in staffing. 

SOURCE: Center responses to standard response categories during interviews. 
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Turning to the technical assistance relationship between the two types of Centers, the 
CCs described the barriers that they saw as impeding their assistance to RCCs, and the RCCs 
reported on barriers in this same relationship from their perspective as the client. 

■ The greatest barriers reported in the RCC-CC technical assistance relationship 
were limitations of the CC’s scope of work and staff capacity, and the issue of 
shifts in state policies or priorities. Four of the 5 CCs reported that “RCCs’ most 
important priorities for assistance fell outside the CCs’ scope of work,” and 2 of 
them called this the greatest barrier (exhibit 3.15). Seven of the 16 RCCs reported 
that this was a barrier, with 4 calling it the greatest barrier. Five of the 16 RCCs 
reported that, “CC staff were not able to spend as much time working with us as 
we would have liked,” and 3 of them named it as the greatest barrier. None of the 
CCs reported this as an impediment to achieving its technical assistance 
objectives with RCCs. For 2 CCs, shifts in state policies or priorities were said to 
impede not only their technical assistance to states but also their technical 
assistance to the RCCs working with states. Both reported this barrier as the 
greatest one they encountered in assisting RCCs. One of the 16 RCCs reported 
this as a barrier to receiving assistance from CCs, but not the greatest barrier. 

■ Four RCCs reported no barriers to receiving technical assistance from CCs. 
However, all 5 CCs did identify barriers (exhibit 3.15). 
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Exhibit 3.15. Barriers to CCs’ assistance to RCCs, 2008-09, perceived by RCCs 
and CCs 


Number of Centers reporting 



Barrier 

Greatest barrier 

Barrier to delivery/ receipt of technical assistance 

RCC 

(N=16) 

CC 

(N=5) 

RCC 

(N=16) 

CC 

(N=5) 

No barriers reported to receiving assistance 

4 


4 

0 

RCCs’ most important priorities for assistance fell 
outside the CCs’ scope of work 

7 

4 

4 

2 

CC staff were not able to spend as much time working 
with RCCs as the RCCs would have liked 

5 

0 

3 

0 

The RCC and CC were unable to develop a productive 
working relationship 

3 

0 

0 

0 

RCCs secured most of the technical assistance they 
needed from other sources 

2 

1 

2 

0 

CCs did not have the expertise RCCs needed 

2 

1 

0 

0 

A policy or priority shift at the state caused the CCs’ 
assistance to RCCs to be less helpful than it might 

1 

2 

0 

2 

There was a lack of coordination or communication 
within an RCC 

1 

2 

0 

0 

RCC staff did not have time to work with CCs 

1 

0 

1 

0 

RCCs experienced turnover in staffing 

0 

1 

0 

0 

RCCs placed a higher priority on completing short- 
term work than on addressing long-term purposes 

0 

1 

0 

0 

The process of negotiating a work scope and 
organizing projects took too long 

0 

0 

0 

0 

RCCs would have preferred to locate and contract 
directly with experts or consultants rather than working 
with the CCs 

0 

0 

0 

0 

RCCs experienced turnover in leadership 

0 

0 

0 

0 


EXHIBIT READS: For the 2008-09 program year, four RCCs reported no barriers to achieving receiving 
technical assistance from CCs. Seven RCCs reported that a barrier to the receipt of technical assistance 
was that the RCCs’ most important priorities for assistance fell outside the CCs’ scope of work. 

SOURCE: Center responses to standard response categories during interviews. 


Quality Assurance Procedures 

Finally, a potentially important aspect of Center operations was the steps taken to ensure 
quality in the technical assistance delivered. Depending on what a Center provided to clients, 
different approaches to quality assurance might be feasible: in particular, materials would lend 
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themselves more readily to formal review before delivery, compared with services. Recognizing 
that there were differences in the mix of resources and activities delivered across projects and 
across Centers, the team gathered Centers’ self-reports on the particular sources of expertise 
brought to bear in quality assurance for each project in the 2008-09 sample, selected from a list 
of possible sources. In addition, the interviews with Center directors addressed their standard 
procedures, if any, for quality assurance. 

■ While most RCC and CC projects were reviewed in-house, many of the CC 
projects also underwent review by external experts and by ED. Review by 
Center staff was reported for more than 90 percent of projects (92 percent of RCC 
projects, 96 percent of CC projects) (exhibit 3.16). Just under half of all projects 
were reviewed by other staff, not regularly employed by the Center, in the lead 
grantee organization (47 percent of RCC projects, 31 percent of CC projects) or in 
subgrantee organizations (38 percent of RCC projects, 58 percent of CC projects). 
For 85 percent of the sampled CC projects, experts were retained to review 
materials; for 81 percent, ED provided a review. The corresponding figures for 
RCC projects were lower, at 44 percent and 17 percent respectively. 


Exhibit 3.16. Sources of expertise used in quality assurance, by Center type, 
2008-09 


Percent of sampled Center projects 



RCC 

CC 

All 

Source 

projects 

(n=93) 

projects 

(n=26) 

projects 

(n=119) 

Internal Center staff (of lead grantee organization 
and/or subgrantees) 

92% 

96% 

93% 

Other staff in the lead organization, not formally 
employed by the Center 

47 

31 

44 

Outside experts retained to review drafts/materials 

44 

85 

53 

Content Center(s) 

39 

15 

34 

Other staff in subgrantee organization(s) 

38 

58 

42 

U.S. Department of Education 

17 

81 

31 


EXHIBIT READS: For the 2008-09 program year, 92 percent of RCC projects reportedly underwent review by 
internal Center staff prior to delivery 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials and coded by the 
evaluation team. 


When Center directors were asked in interviews whether they had “a formal process” for 
“quality assurance in technical assistance during 2008-09,” all but 1 of the 21 (an RCC director) 
said yes. The specific procedures Center directors reported using for quality assurance differed 
for products and services. Across the 14 Centers who described their in-house product reviews 
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(10 of the 16 RCCs and 4 of the 5 CCs), Centers explained the two types of pre-release reviews 
conducted for the products they distributed in print or electronic form. Most often, in-house 
reviews targeted formatting and style, such as copy editing (7 RCCs and 3 CCs). Centers also 
reported using one or more of the following sources for conducting substantive internal reviews 
(9 RCCs and 3CCs): knowledgeable Center staff members, Center directors personally reviewing 
all products, or staff members from the grantee or subgrantee organization who were not on the 
Center staff. 

For services, a detailed review prior to delivery would not be feasible in the same way 
that it would for a product, and Centers instead described cycles of in-house review informed by 
evaluative feedback from clients. As a part of these cycles of review, client feedback was 
gathered systematically by internal or external evaluators or gathered by the technical assistance 
providers in service reviews with clients. A total of 10 Centers (9 RCCs and 1 CC) cited client 
feedback (e.g., event evaluations) as a source of input to quality assurance. 
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4. State Managers’ Assessment of Center 
Technical Assistance 


The Centers were expected to provide technical assistance that would build states’ 
capacity to carry out their responsibilities under NCLB, supporting district and school efforts to 
close achievement gaps and raise student achievement. Thus, the perceptions of Center 
assistance expressed by state agency managers provide a relevant perspective on the outputs of 
the overall Center program. These managers were in a position to view the planning, operations, 
and state agency perceptions of the technical assistance provided by both the RCCs and the CCs. 
This evaluation inquired into the extent to which Center technical assistance served state 
purposes and, according to state managers, built state capacity. 

This chapter examines how the work of the Centers was regarded by senior managers in 
state education agencies, addressing the following research questions: 

■ What was the performance of the Comprehensive Centers in addressing state 

needs and priorities? How did their performance change over the period of time 
studied? 

■ How did the Centers’ state clients define their needs and priorities? 

■ To what extent, as reported by states, did Center assistance expand state 
capacity to address underlying needs and priorities and meet the goals of 
NCLB? 

■ To what extent did states rely on other sources of technical assistance 
besides the Centers? What were other sources of technical assistance that 
states used? How did the usefulness of Center assistance compare with the 
usefulness of assistance from other sources? 

The analysis in this chapter draws on information gathered through the survey of state 
managers administered in each of the three data collection year as well as interviews with Center 
staff conducted by the evaluation team . 50 


Extent to Which Center Assistance Served State Purposes 

ED placed a significant emphasis on having the Centers, both RCCs and CCs, deliver 
technical assistance that would advance state efforts to implement NCLB. In developing their 
work plans and delivering technical assistance, the Centers were expected to target their work on 


50 A detailed description of the changes to the state manager survey between the 2006-07 and 2007-08 
administration cycles as well as copies of both survey instruments can be found in appendix E. Supplemental tables 
related to this chapter are in appendix F. 
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state concerns and priorities. States’ views of how well the assistance met their own purposes 
provided one perspective on the success of the program. 

All state managers responding to the survey indicated the extent to which Center 
technical assistance had served their state purposes, in each year of the study. Further elaboration 
was provided by those who reported that this assistance had served state purposes less than 
completely: 


■ More state managers in each year reported that Center technical assistance 

addressed their states’ purposes. The percentage reporting that Center technical 
assistance had “served the state’s purposes completely” was 36 percent for 2006- 
07, 47 percent for 2007-08, and 56 percent for 2008-09 (exhibit 4.1). 51 


Exhibit 4.1. Extent to which technical assistance from the Centers served state 
purposes, as judged by state managers, by year 



It did not serve the state's 
purposes 

It was a good start, but some 
important priorities were 
not addressed 


It was a good start 


It served the state's 
purposes completely 


2006-07 2007-08 2008-09 

(n=56) (n=57) (n=54) 


EXHIBIT READS: For the 2006-07 program year, 36 percent of state managers, weighted, reported 
that Center technical assistance served the state’s purposes completely. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally 
represented in instances where more than one manager from a state responded. 


5 1 The survey of state managers was administered to managers in a census of all state departments of education. 
Therefore, difference in the distribution of state responses to the survey across time is presented without inferential 
statistics or assessments of statistical significance. 
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■ Over the three years, state managers reported greater interest in seeking more 
technical assistance from Centers. The survey posed a series of follow-up 
questions asking why, to the respondents who did not report that Center assistance 
“served the state’s purposes completely” (exhibit 4.2). These state managers 
were asked to check off as many as eight possible reasons that the assistance was 
less helpful to their state than it might have been. In 2008-09, the most frequent 
response among the subgroup of state managers whose purposes were not 
completely served was, “Center staff are not able to spend as much time working 
with the state as we would like.” The frequency of this response increased over 
time: it was selected by 17 percent of these respondents in 2006-07, 27 percent in 
2007-08, and 43 percent in 2008-09. 53 (These percentages of the subgroup 
represented 9, 10, and 16 percent of all state managers, weighted, respectively.) 

■ State managers’ reservations about Center technical assistance were not the 
same issues that Center directors saw as barriers to meeting their aims. As 
discussed in chapter 3, the Center directors were asked to indicate what had 
impeded their assistance to states in 2008-09, choosing from a list that included 
all the responses that also appeared on the state manager survey. For the 20 state 
managers (weighted) reporting Center assistance had served the state’s purposes 
less than completely, the issue reported by the highest percentage of state 
managers (43 percent) was that “Center staff are not able to spend as much time 
working with the state as we would like” while the top issue for Center directors 
(48 percent) was that “a policy or priority shift at the state level caused the 
Center’s assistance to be less helpful than it might be” (exhibit 4.2). 


52 In each reporting year, a small number of state managers who indicated they were not fully satisfied with the 
technical assistance they received from the Centers failed to answer the follow-up question asking why they were 
less than satisfied. In 2006-07, 64 percent of state managers were less than fully satisfied and 53 percent provided 
responses to the follow-up question. These percentages were 53 percent and 37 percent for 2007-08 and 44 percent 
and 37 percent for 2008-09. 

53 The percentage of state managers reporting that their state’s purposes were not completely served varied by year. 
Thus, for the follow-up question (“reasons the technical assistance has been less helpful than it might be”), 
comparison of percentages from year to year may include variation in responses over time as well as changes in 
respondents addressing the question. 
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Exhibit 4.2. Reasons why Center assistance served some states’ purposes less 
than completely, by year, as reported by state managers, and 
barriers reported by Center directors to achieving Center aims, 2008- 
09 



Percent of those state managers 
(weighted) who reported Center 
assistance had not served their 
purposes completely 

Percent of all 
Center directors 
reporting 
barrier 

Reasons/Barriers 

2006-07 

(Total n=30) 

2007-08 

(Total n=21) 

2008-09 

(Total n=20) 

2008-09 

(N=21) 

Center staff are not able to spend as much time 
working with the state as we would like 

17% 

27% 

43% 

5% 

The state secures most of the technical 
assistance it needs from other sources 

31 

24 

27 

14 

The process of negotiating a work scope and 
organizing projects takes too long 

33 

31 

25 

5 

The state's most important priorities for 
assistance fall outside the Center’s scope of 
work 

33 

35 

22 

24 

The Center does not have the expertise the state 
needs 

15 

31 

18 

0 

The state has been unable to develop a 
productive working relationship with the Center 

3 

15 

15 

5 

The state would prefer to locate and contract 
directly with the experts or consultants from 
whom it needs assistance, rather than working 
through the Centers 

19 

36 

5 

14 

A policy or priority shift at the state level caused 
the Center’s assistance to be less helpful than it 
might 

7 

13 

4 

48 


NOTE: The first three columns display the percentage of state managers (weighted so that each state was equally 
represented) reporting reasons the technical assistance has been less helpful to the state than it might have been. 
Based on Center director interview responses, the last column on the far right displays the percentage of Center 
directors who reported each to be a barrier to the Center achieving its aim. 

EXHIBIT READS: For the 2006-07 program year, of the 30 state managers, weighted, reporting Center assistance 
served their state's purposes less than completely, 17 percent said a reason was that the Center staff were not able 
to spend as much time working with the state as they would like. 

SOURCE: Survey of State Managers and Center director responses to standard response categories during a phone 
interview. State respondents limited to those who did not answer “It served our purposes completely” to the preceding 
question. Responses to the state manager survey weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Reported Assistance Delivery and Capacity Building for State 
NCLB Responsibilities 

The primary aim of the Center program was to deliver technical assistance that would 
both address state priorities in implementing NCLB and build state capacity for NCLB 
implementation. Senior state managers reported their perceptions of the extent to which the 
Centers had delivered assistance and built capacity in the areas of NCLB responsibility that were 
technical assistance priorities for their state. 

■ State managers consistently placed a priority on assistance with statewide 
systems of support or school support teams. When asked about state 
responsibilities with which they wanted technical assistance from any source (not 
just from the Centers), more than 90 percent of managers, weighted, (95 percent 
in 2007-08 and 94 percent in 2008-09) reported these functions to be a major or 
moderate priority for assistance in 2007-08 and 2008-09 (exhibit 4.3). In 2006-07, 
with a differently worded survey question, the responses were generally 
consistent: 98 percent reported that “building or managing a statewide system of 
support for districts and schools identified for improvement under NCLB” was a 
major or moderate priority for assistance, and 90 percent said the same for 
“training or managing school support teams.” 

■ The other leading areas of priority for technical assistance were policies and 
practices for English language learners; state assessment and accountability 
systems; and research-based curriculum, instruction, or professional 
development in academic subjects. For each area, between 73 and 74 percent of 
state managers in 2007-08 and between 75 and 79 percent of state managers in 
2008-09 identified these areas of state responsibility as a major or moderate 
priority for technical assistance, (exhibit 4.3). 

■ State managers reported that Centers delivered technical assistance in an area 
they had identified to be a major or moderate state priority for technical 
assistance. This analysis focused on those managers who reported that their state 
prioritized technical assistance with a particular NCLB responsibility. Looking at 
the top four priorities for technical assistance reported by state managers, in 2008- 
09 at least 66 percent of those managers, weighted, who named a particular 
priority also reported that the Centers had delivered assistance with that area (at 
least 77 percent in 2007-08; exhibit 4. 3). 54 For state systems of support and school 
support teams, the most widely reported priority across states, the percentage 
exceeded 90 percent in each year (94 percent in 2007-08 and 91 percent in 2008- 
09). The exceptions to the overall pattern of assistance delivery were found in the 
two least widely reported priorities, supplemental service and choice provisions 


54 The subgroup of state managers who identified a particular area of state responsibility to be a major or moderate 
priority for technical assistance varied by year. Thus, for the follow-up question about the priority areas in which 
states received technical assistance from Centers, comparison of the percentages from year to year may include 
variation in responses over time as well as changes in respondents addressing the question. 
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and communication with parents, where the percentages reporting help from the 
Centers were below 55 percent in both 2007-08 and 2008-09 (exhibit 4.3). 

■ State reporting of Center’s delivery of technical assistance across state priorities 
under NCLB declined from 2007-08 to 2008-09. With just one exception, 
communication with parents or the public, the state managers who placed at least 
a moderate priority on a particular area of NCLB state responsibility were less 
likely to report Center help with that responsibility in 2008-09 than in the 
previous year. The three largest drops from 2007-08 to 2008-09 were in the 
reports that Centers had provided assistance with educators’ use of assessment 
data (from 83 percent to 58 percent); research-based curriculum, instruction, or 
professional development in academic subjects (from 88 percent to 72 percent); 
and monitoring compliance with NCLB requirements (from 80 percent to 64 
percent) (exhibit 4. 3). 55 

The drop in NCLB -related technical assistance was not accompanied, however, by an 
overall report that the Centers were failing to address state purposes. As discussed above, the 
weighted proportion of state managers reporting that Center assistance had served state purposes 
“completely” rose from 47 percent to 56 percent over this time (exhibit 4.1). As well, more state 
managers reported at least moderate contributions of Center technical assistance to increasing 
their state’s capacity in areas identified as state priorities: 

■ State managers’ perceptions of the Centers’ contributions to state capacity 
increased from 2007-08 to 2008-09 in the four NCLB-related areas of highest 
priority. Again, the focus here is on those managers who reported a particular 
responsibility as a technical-assistance priority. When asked to rate the extent to 
which Center assistance had expanded the state’s capacity to carry out state 
NCLB responsibilities, a higher percentage of those state managers who had 
identified each area of responsibility as a priority credited the Centers with great 
or moderate contributions to state capacity in 2008-09 than in 2007-08 in 
statewide systems of support (72 percent in 2007-08 and 82 percent in 2008-09), 
English language learners (59 and 73 percent), state assessment and 
accountability systems (57 and 59 percent), and research-based curriculum, 
instruction, and professional development (64 and 77 percent) (exhibit 4.4). 56 


55 Percentages are based on the state manager respondents who identified a particular area of state responsibility to 
be a major or moderate technical assistance priority, which varied by year. Thus, for the follow-up question about 
the priority areas in which states received technical assistance from Centers, comparison of the percentages from 
year to year may include variation in responses over time as well as changes in respondents addressing the question. 
5h The subgroup of state managers who identified a particular area of state responsibility to be a major or moderate 
priority for technical assistance varied by year. Thus, for the follow-up question about state reported capacity 
building, comparison of the percentages from year to year may include variation in responses over time as well as 
changes in respondents addressing the question. 
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Exhibit 4.3. State priorities for technical assistance and assistance received from 
Centers, 2007-08 and 2008-09 


Percent of state managers (weighted) 

2007-08 

2008-09 

Area of state responsibility 
under NCLB 

Reporting 
responsi- 
bility as a 
major or 
moderate 
priority 

Responsibility 
is major or 
moderate 
priority and 
Center technical 
assistance 
received in that 
area 

Reporting 
responsi- 
bility as a 
major or 
moderate 
priority 

Responsibility 
is major or 
moderate 
priority and 
Center technical 
assistance 
received in that 
area 

Statewide systems of support 
or school support teams 
(n=56; n=52) 

95% 

94% 

94% 

91% 

Policies and practices for 
English language learners 
(n=55; n=51) 

74 

79 

79 

76 

State assessment and 
accountability systems 
(n=53; n=50) 

74 

77 

79 

66 

Development or dissemination 
of research-based curriculum, 
instruction, or professional 
development programs in 
academic subject(s) 

(n=54; n=51) 

73 

88 

75 

72 

Assistance with educators’ use 
of assessment data 
(n=54; n=51 ) 

68 

83 

69 

58 

Monitoring compliance with 
NCLB requirements 
(n=53; n=51) 

64 

80 

59 

64 

Administering supplemental 
educational services (SES) and 
choice provisions 
(n=54; n=50 ) 

49 

52 

48 

43 

Communication with parents or 
the public 
(n=54; n=50) 

47 

41 

49 

54 


NOTE: Percentages are based on the state manager respondents who rated each area a major or moderate 
technical assistance priority, which varied by year. 

EXHIBIT READS: For the 2007-08 program year, 95 percent of state managers, weighted, reported that the NCLB 
area of state responsibility of statewide systems of support or school support teams was a major or moderate priority 
for technical assistance for their state. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Exhibit 4.4. Extent to which Center assistance expanded state capacity in 

priority areas, as judged by state managers who rated the area as 
a major or moderate technical assistance priority, by year 



Percent reporting 
capacity expanded by a 
great or moderate extent 

Area of state responsibility under NCLB 

2007-08 

2008-09 

Statewide systems of support or school support teams 
(n=56, n=50) 

72% 

82% 

Policies and practices for English language learners 
(n=43, n=40) 

59 

73 

State assessment and accountability systems 
(n=42, n=39) 

57 

59 

Development or dissemination of research-based 
curriculum, instruction, or professional development 
programs in academic subject(s) 

64 

77 

(n=41 , n=39) 

Assistance with educators’ use of assessment data 
(n=37, n=36) 

62 

61 

Monitoring compliance with NCLB requirements 
(n=35, n=30) 

57 

57 

Administering supplemental educational services (SES) 
and choice provisions (n=25, n=26) 

Communication with parents or the public 
(n=25, n=26) 

44 

39 

48 

50 


NOTE: Percentages are based on the state manager respondents who rated each area a major or moderate 
technical assistance priority, which varied by year. State managers who chose the response, “Does not 
apply, or not able to judge,” were included in the denominator of the percent calculation. 

EXHIBIT READS: For the 2007-08 program year, among the 56 state managers (weighted) who reported 
that technical assistance in statewide systems of support or school support teams was a major or moderate 
priority for their state, 72 percent reported that technical assistance received from the Centers expanded the 
state’s capacity in this area to a great or moderate extent. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in 
instances where more than one manager from a state responded. 


While the analyses just described focus on reports of technical assistance delivery and 
capacity building in the specific areas of state responsibility that particular state managers 
identified as priorities, it is also useful to know the extent to which managers in all states said 
technical assistance from the Centers had expanded state capacity, irrespective of the priority the 
state accorded to that area. 

■ The percent of all state managers, weighted, reporting that assistance from the 
Centers had increased their state’s capacity to a “great” extent in at least one 
area of state NCLB responsibilities increased from 62 percent in 2007-08 to 75 
percent in 2008-09. The percent reporting either a “great” or a “moderate” 
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expansion of capacity from Center assistance in at least one area rose from 85 
percent to 96 percent. 

Capacity building at the state level can take many forms, and to help illustrate some of 
them the textbox below provides a summary from case studies, “ Illustrative Examples of 
Reported Capacity Building Activities.” A description of this data collection is included in 
Appendix E. 


Illustrative Examples of Reported Capacity Building Activities 

As a supplementary data collection, we interviewed SEA staff in 10 states and relevant Center staff 
about the technical assistance provided on SSOS, the most common area for technical assistance. 
The states were selected to maximize the number of Centers included (a total of 10 RCCs and 3 
CCs) and to represent the full range of SEA reported experiences (i.e. , cases include SEAs whose 
staff reported capacity was expanded “to a great extent,” “to a moderate extent,” and “to a small 
extent”). The interviews elicited information about the capacity building support provided by the 
Centers and examples of what SEA staff perceived as the benefits of such assistance. The 
interviews focused specifically on three potential capacity building areas: i) technology, systems 
and infrastructure improvements; ii) SEA staff knowledge/skill enhancements; iii) access to external 
support/expertise. The following illustrative examples are not intended to be representative of all 
Center work and provide no indication of the extent to which the following descriptions are 
representative of all states. 

Although SEA respondents commonly highlighted Center support with tool development and use, 
resulting assessments of these efforts varied. Products developed through Center assistance 
included things such as on-line application for districts to submit annual improvement plans that 
SEA staff could access for monitoring purposes, a data system with tools to facilitate district use of 
data for needs-assessment purposes, and a web-based store of resources, training materials, 
research materials, and other SSOS information. A couple of the interviewed SEAs reported that 
the developed tools improved SEA efficiency, while other SEAs reported the tools as useful but 
less integral to subsequent SEA activities in part because staff were not fully implementing the 
tools. 

Working with Centers to refine state processes for supporting districts and schools in need of 
improvement was another Center activity highlighted by some interviewed SEAs. This work 
included things like establishing district and school support teams and training the support 
providers. SEAs also reported working with the Centers to help improve inter-departmental 
communication and coordination through Center facilitated meetings. 

Finally, Centers reported providing SEA staff with knowledge and skills in order to potentially build 
SEA capacity. Such activities included access to synthesized research, best practices, policy briefs, 
and innovative approaches or SSOS models used by other states. Some SEA staff reported 
participation in RCC-led regional professional development workshops. 


State Uses and Perceptions of the Centers Compared with 
Other Sources of Technical Assistance 

With many other sources available to states, the Centers were designed to fill niches — in 
particular, that of building capacity for state implementation of NCLB requirements — rather than 
addressing every purpose for which states might seek technical assistance. The responses of state 
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managers confirmed that the states relied upon multiple sources of technical assistance in their 
practice in all three years and that they used these sources for different purposes. 

■ Among these state managers, the Centers were the source most often reported 
to be used to a “great” or “ moderate ” extent. More than 80 percent named the 
Centers as a technical assistance resource used at least to a moderate extent in 
both years, 2007-08 and 2008-09 (exhibit 4.5). More than three-quarters also 
cited their Regional Educational Laboratory as at least a moderate source in 
2007-08 (76 percent) and professional associations in 2008-09 (79 percent). In 
2006-07, with a differently worded question, the Centers were also ranked high 
as an assistance source: they were one of the top three sources, along with 
professional associations and Regional Educational Laboratories (appendix 
exhibit F.5). 


Exhibit 4.5. States’ use of external sources of technical assistance, by year 



Percent of state managers 
(weighted) reporting to a great 
or moderate extent: 

Source of technical assistance 

2007-08 

(n=57) 

2008-09 

(n=50) 

Comprehensive Center network 

87% 

82% 

Regional Educational Laboratory 

76 

74 

U.S. Department of Education 

64 

66 

Professional associations 
(e.g., CCSSO, ASCD) 

64 

79 

Senior managers in other SEAs 

59 

64 

Consulting firms or private 
contractors 

56 

51 

Colleges and universities 

46 

52 


EXHIBIT READS: For the 2007-08 program year, 87 percent of state managers, 
weighted, said that they relied on the Centers as a source of technical assistance to a 
great or moderate extent. 

SOURCE: Survey of State Managers. Responses weighted so that each state was 
equally represented in instances where more than one manager from a state 
responded. 

State managers’ reports of the purposes for which they used each source of technical 
assistance shed light on the niches occupied by the Center program — the purposes for which the 
Centers were the most commonly used source. These responses also revealed some purposes for 
which other sources were the most widely used in particular years. 

■ Across different purposes for technical assistance, state managers identified the 
Centers as the leading source for planning, completing tasks, and developing 
skills. In 2008-09, the Centers were cited as the top source “to plan the initial 
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steps in solving a problem” (59 percent), to complete tasks that either “the state 
could do itself if it had more staff or resources” (64 percent) or “for which the 
state lacks expertise” (53 percent), and “to develop the skills of SEA or 
intermediate education agency staff’ (55 percent). While not the top source, the 
Centers were cited second to professional associations as a resource states used 
“to gather information or to keep current with new ideas” (73 percent compared 
with 79 percent) (exhibit 4.6). 


Exhibit 4.6. Purposes for which states used external sources of technical 
assistance, 2008-09 


Purpose in seeking technical assistance 
(percent of state managers) 

(n=54) 


External Source 



Comprehensive Center network 

73% 

59% 

64% 

53% 

55% 

32% 

2% 

Regional Educational Laboratory 

66 

44 

34 

38 

30 

19 

9 

U.S. Department of Education 

58 

43 

14 

18 

23 

7 

11 

Professional associations (e.g., 
CCSSO, ASCD) 

79 

23 

20 

20 

33 

9 

4 

Senior managers in other SEAs 

63 

36 

7 

7 

14 

5 

10 

Consulting firms or private 
contractors 

29 

31 

50 

42 

25 

32 

21 

Colleges and universities 

50 

29 

26 

21 

19 

25 

20 


EXHIBIT READS: For the 2008-09 program year, 73 percent of state managers (weighted) said that they turned to 
the Centers to gather information or to keep current with new ideas. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 


■ Over time, the Centers were reported by state managers as the top resource for 
more purposes. In 2006-07, state managers reported Centers as the top source in 
two areas: for planning the initial steps in solving a problem (66 percent) and 
developing skills of SEA staff (61 percent). In the following years (2007-08 and 
2008-09), Centers were reported as the top source for two additional purposes: to 
help states complete tasks where they lacked resources (58 percent and 64 
percent, respectively) or expertise (49 percent and 53 percent, respectively). 
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State managers also provided a rating of the usefulness of Center technical assistance, in 
comparison with other sources, with respect to particular areas of NCLB state responsibility. 

■ In the area of statewide systems of support or school support teams, a majority 
of state managers (62 percent, weighted) called the Centers’ technical 
assistance “ more useful” than assistance from other sources in 2008-09 (exhibit 
4.7). This was the only area in which more than 80 percent of state managers said 
they could make a judgment on usefulness. The pattern of responses was similar 
in 2007-08 (appendix exhibit F.9). 


Exhibit 4.7. Usefulness of Center assistance compared with 
assistance from other sources, 2008-09 


Percent of state managers, weighted, rating 
Center technical assistance as: 

Areas of state responsibility under NCLB 

More 

useful 

About the 
same 

Less 

useful 

Not able to 
judge 

Statewide systems of support or school support 
teams (n=49) 

62% 

21% 

4% 

13% 

Policies and practices for English language 
learners (n=46) 

35 

29 

4 

32 

State assessment and accountability systems 
(n=41) 

23 

28 

6 

42 

Development or dissemination of research-based 
curriculum, instruction, or professional development 
programs in academic subject(s) (n=45) 

41 

35 

2 

22 

Assistance with educators’ use of assessment data 
(n=41) 

28 

37 

14 

21 

Monitoring compliance with NCLB requirements 
(n=38) 

43 

19 

7 

32 

Administering supplemental educational services 
(SES) and choice provisions (n=32) 

16 

16 

11 

58 

Communication with parents or the public (n=34) 

27 

25 

3 

45 


NOTE: States that chose the response, “not applicable, state has not sought assistance for this purpose,” were 
excluded from the analysis. 

EXHIBIT READS: For the 2008-09 program year, 62 percent of state managers (weighted) reported that Centers 
were more useful than other sources of technical assistance for the state NCLB responsibility of statewide systems of 
support or school support teams. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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SEA Overall Rating of Center Assistance 


As senior officials in their agencies, the state managers were in a position to provide 
overall judgments of the Center technical assistance their agency had received. The state 
manager survey included scales for the degree to which Center technical assistance, in general 
(not specific to a particular project), was seen as relevant and useful, comprising five indicators 
of relevance and five indicators of usefulness. Ratings were based on a 1 to 5 scale. 

■ On a scale of 1 ( very low degree ) to 5 (very high degree), on average, state 
managers rated the Centers’ technical assistance about a 4 (“high” degree) on 
relevance and usefulness in each year. The average relevance scores in each year 
were 3.94 in 2006-07, 3.92 in 2007-08, and 4.07 in 2008-09, and the average 
usefulness scores were 3.86 in 2006-07, 3.99 in 2007-08, and 4.21 in 2008-09. 59 

■ The 2008-09 ratings of relevance and usefulness from the survey of state 
managers suggest that Centers were targeting their assistance more closely on 
key priorities and providing resources that states could continue using. Among 
the individual components of the relevance index, state managers gave the highest 
average rating of 4.30 for Center assistance addressing “an important state 
priority” in 2008-09. On the usefulness index, the highest average rating of 4.40 
in 2008-09 was for Centers’ having “provided state staff with information or 
resources that they will use again” (exhibit 4.8). 


57 The survey items in the relevance and usefulness scales were reduced in number between the 2006-07 and 2007- 
OS surveys. Because no distinct sub-dimensions existed in the 2006-07 scales, the scales were simplified to reduce 
burden. Items were selected for deletion based on a detailed analysis of the impact of each item on the overall 
reliability of the scale. In 2007-08 the reliability of the revised relevance scale was .926 (Cronbach's Alpha) and that 
of the revised usefulness scale was .911. 

58 The response options were: 5 (to a very high degree), 4 (to a high degree), 3 (to a moderate degree), 2 (to a low 
degree), and 1 (to a very low degree). 

59 The survey from 2006-07 was revised for the 2007-08 administration (and remained unchanged for 2008-09), so 
the scores are not directly comparable. The specific changes in the survey are described in appendix E. 
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Exhibit 4.8. Mean ratings of the relevance and usefulness of Center 

assistance by component, as judged by state managers, 2008-09 


Component 

Ratings among 
state managers 
(weighted) 

Relevance 

4.07 

Addressed an important state priority 

4.30 

Addressed a need or problem that the state faces 

4.23 

Provided information, advice, and/or resources that could be used to 
guide decisions about policies, programs, and practices 

4.09 

Addressed a challenge that the state faces related to the implementation 
of NCLB 

3.94 

Addressed the state’s specific challenges (e.g., policy environment, 
leadership capacity, budget pressures, local politics) 

3.87 

Usefulness 

4.21 

Provided state staff with information or resources that they will use again 

4.40 

Was timely 

4.25 

Helped state staff to develop skills that they will be able to exercise again 

4.11 

Helped the state to solve a problem 

4.05 

Helped the state to maintain or change a policy or practice 

3.89 


EXHIBIT READS: For the 2008-09 program year, state managers (weighted) gave a mean relevance rating 
of 4.07 across all specific rating items, and a mean of 4.30 when rating whether the assistance their state 
had received from the Centers addressed an important state priority. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in 
instances where more than one manager from a state responded. 


72 



5. Variation in the Reported Quality, Relevance, and 
Usefulness of the Centers’ Technical Assistance 


While state managers’ reports provide important input to an assessment of the overall 
work of the Center program (chapter 4), additional perspectives are helpful in assessing the 
quality of the technical assistance projects and their relevance and usefulness to the offices and 
the teams of professionals directly served. As the two types of Centers played distinct roles in 
providing technical assistance, ratings were examined for the Center program and for each center 
type in the second through the fourth program years (2006-07, 2007-08, 2008-09). A deeper look 
at variation across Centers and projects and across program years might also provide information 
for program improvement: additional support or oversight might be provided if quality, 
relevance, or usefulness appeared substantially weaker in particular groups of Centers or 
projects, or for particular types of participants. Thus, variation was investigated by features of 
project design (project scope and the type of activities offered) and by participant experiences in 
the project as well as participant background. 

This chapter analyzes the ratings of quality, relevance, and usefulness given to the Center 
projects sampled for the evaluation in each year, addressing the following research questions: 

■ To what extent was the assistance provided by the Centers of high quality, high 
relevance, and high usefulness? 

■ Did the quality, relevance, or usefulness of Center assistance change over 
the period of time studied? 

■ What was the variation in the quality, relevance, and usefulness of Center 
assistance across types of projects and participants? 

Quality was judged on three items called dimensions; relevance was assessed with eight survey 
items and usefulness with 11 items (exhibit 5.1). Sampled projects were rated for technical 
quality by panels of experts with strong knowledge of the content or substantive focus of the 
specific projects they reviewed. 60 Ratings of the sampled projects’ relevance and usefulness 
were gathered by surveying a sample of participants — state staff, intermediate agency staff, local 
educators working on behalf of the state, and RCC staff — who were the intended beneficiaries of 
the project and had received at least some of the technical assistance it provided. 61 Participant 
ratings focused on the specific project in which respondents had participated whereas the state 
managers rated the relevance and usefulness of the overall Center assistance that the state had 
received. 62 Each overall measure (quality, relevance, or usefulness) was calculated as the mean 


60 See chapter 2 for more information about the data sources and procedures for gathering, coding, and analyzing the data 
reported in this chapter, including discussions of the methods used for expert ratings and participant surveys. 

61 Copies of the expert panel review rating materials and the project participant surveys can be found in appendix G. 
Supplemental tables related to this chapter are in appendix H. 

62 Additionally, the state manager survey items for the relevance and usefulness scales were reduced in number 
between the 2006-07 and 2007-08 surveys. Thus, measures of relevance and usefulness for the state managers were 
different from the measure administered to project participants. 
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of ratings assigned to each item. The item-level ratings themselves were based on 5-point rating 
scales (exhibit 5. 2). 63 

The chapter describes patterns observed in the ratings across the three program years. 
However, changes over time are presented with caution given a number of limitations or caveats 
related to the ratings of quality, relevance, and usefulness. Each year’s sample of projects was a 
purposive one, with unknown differences in the characteristics of the projects across years. For 
the quality ratings, while the same procedures were followed in each year, the cadre of panelists 
was not entirely unchanged. 64 Further, the Centers selected and submitted the specific project 
materials that the experts reviewed, and Center staff became aware of the evaluation procedures 
and criteria during 2007. In the first (2006-07) rating cycle, Centers were given instructions in 
May 2007, within two months of the end of the program year, for assembling materials. In the 
subsequent cycles, Centers knew what they would be asked to submit and thus may have 
gathered and prepared materials during the year as projects progressed. Similarly, for the 
relevance and usefulness ratings, the Centers learned in 2007 that they would be asked to provide 
full lists of project participants, from which the evaluation team would select survey respondents, 
and thus they had the opportunity to compile the needed records during the 2007-08 and 2008-09 
program years. Finally, the participants surveyed were a different group of individuals in each 
year, sampled from all participants in the selected projects. 


63 Efforts were made to develop parallel wording and rubrics that would result in similar gradations between rating 
levels (e.g., very high vs. high vs. moderate) across the three measures. However, given the different content of each 
set of items within the three measures and the different contexts for the ratings (experts who underwent training for 
the rating process and reviewed identical packages of materials vs. survey respondents who typically participated in 
different subsets of project activities), the ratings across the three measures are not directly comparable. 

64 Sixty-six of the 70 panelists in 2006-07 returned for the 2007-08 rating cycle, and 14 new panelists were added. 
Seventy-eight of the 80 panelists who reviewed materials in 2007-08 returned for 2008-09. 
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Exhibit 5.1. 


Quality, relevance, and usefulness items 


From expert panel scoring From project participant surveys 65 

Technical quality Relevance Usefulness 


Reviewers were directed to 
assign a score to each 
dimension and to include the 
basis for their ratings on the 
rating form, including the 
specific artifacts on which their 
score was based. The three 
dimensions are: 

a. Demonstrated use of the 
appropriate documented 
knowledge base-to include 
an accurate portrayal of the 
current state of information 
with prominence to those 
with the most 

accurate/rigorous evidence 

b. Fidelity of application of the 
knowledge base to the 
products and services 
provided-materials are 
consistent with the 
best/accurate information 
available and the 
presentation adequately 
conveys the confidence of 
the information 

c. Clear and effective delivery- 
information is well organized 
and written and accessible to 
the intended audience for 
easy use 


Based on your experience, to 
what degree was this set of 
activities and resources relevant 
to your work, in each of the 
following respects? 

a. Addressed a need or problem 
that my organization faces 

b. Addressed an important 
priority of my organization 

c. Addressed a challenge that 
my organization faces related 
to the implementation of NCLB 

d. Provided information, advice, 
and/or resources that could be 
directly applied to my 
organization’s work 

e. Addressed our particular state 
context 

f. Addressed my organization’s 
specific challenges (e.g., 
policy environment, leadership 
capacity, budget pressures, 
local politics) 

g. Provided information, advice, 
and/or resources that could be 
used to guide decisions about 
policies, programs, or 
practices 

h. Highlighted the implications of 
research findings (or 
information about best 
practice) for policies, 
programs, or practices 


Based on your experience, to what 

degree was this set of activities 

and resources useful to you, in 

each of the following respects? 

a. Provided resources that were 
easy to understand and easy to 
use 

b. Employed an appropriate format 
(e.g., a work group, a 
conference, individual 
consultation, written products) 

c. Provided adequate opportunity 
to learn from colleagues in other 
states 

d. Included adequate follow-up to 
support the use of new 
information and resources 

e. Were timely 

f. Helped my organization solve a 
problem 

g. Helped my organization 
maintain or change a policy or 
practice 

h. Helped my organization take the 
next step in a longer-term 
improvement effort 

i. Provided my organization with 
information or resources that we 
will use again 

j. Helped my organization develop 
a shared expertise or knowledge 
base 

k. Helped individuals in my 
organization to develop skills 
that they will use again 


65 Participant surveys focused on the specific project in which respondents had participated in contrast to the state 
manager surveys that asked about overall Center assistance the state had received. In addition, the participant survey 
items measuring relevance and usefulness were different from those administered to the state manager survey items 
as the state manager survey items were reduced in number between the 2006-07 and 2007-08 surveys. 
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Exhibit 5.2. Scales for rating of quality by expert panels and relevance 
and usefulness by participants 


Rating 

Quality ratings by content 
area experts 

Relevance or usefulness 
ratings by project 
participants 

5 

Very high — All or almost all of the 
project meets the given indicators 
for a dimension 

To a very high degree 

4 

High — Most of the project meets 
the given indicators for a dimension 

To a high degree 

3 

Moderate — Some of the project 
meets the given indicators for a 
dimension 

To a moderate degree 

2 

Low — Limited parts of the project 
meet the given indicators for a 
dimension 

To a low degree 

1 

Very low — None or almost none of 
the project meets the given 
indicators for a dimension 

To a very low degree 


Average Ratings by Experts and Participants 

Aggregated to the entire program, the ratings of expert panelists and project participants 
provide insight into the quality, relevance, and usefulness of Center assistance. Trends in the 
ratings of sampled projects over time may also be indicative of trends in overall Center technical 
assistance. Thus the data gathered on individual projects were compiled to produce mean ratings 
of quality, relevance, and usefulness for the sampled projects across the program as a whole. In 
these averages, each Center contributed equally to the overall mean for the program . 66 Given the 
different roles assigned to the RCCs and CCs, it is appropriate to examine whether there were 
differences in ratings between the two types of Centers . 68 In addition, any consistent variation in 
the quality, relevance, and usefulness across individual Centers would be useful to understand. 


66 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 
to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 
and each project sampled from a Center contributed equally to the Center mean. 

67 ED structured the Comprehensive Centers program to consist of 5 Content Centers charged with specializing in 
activities related to specific content areas and 16 Regional Centers charged with specializing in interactions with the 
states in a region. 

68 For analyses of ratings presented in this chapter with the project or Center as the unit of analysis, the evaluators 
used a metric derived from Cohen (1988) estimating Cohen's d (an estimate of the effect size defined as the 
differences in means divided by the pooled standard deviation). Adopting the logic of Cohen for what is a moderate 
difference, we adopted a difference in the means of one-half of one standard deviation ( analogous to an effect size 
of .5) as our minimum threshold for highlighting differences. The “pooled standard deviation” for each computation 
varied with the unit of analysis. For analyses conducted at the Center level, the pooled standard deviation was 
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■ Center technical assistance, overall, was rated higher on each measure in each 
successive year, with 2008-09 ratings between “moderate” and “high” for 
quality, and “high” for relevance and usefulness ( exhibit 5.3). On a scale of 1 to 
5 with a 3 representing “moderate” and a 4 representing “high,” the program-wide 
average rating for the sampled projects rose from 3.34 in 2006-07, to 3.51 in 
2007-08, to 3.57 in 2008-09 for technical quality (scored by panels of content 
experts). Over this same period of time, Centers’ average rating for relevance 
(scored by participants) went up from 3.94 in 2006-07 to 4.08 in 2007-08 and 
4.15 in 2008-09. The greatest changes in participant ratings of relevance were in 
the items related to the tailoring of assistance-addressing the specific challenges 
or particular context of the SEA, or providing information or resources that could 
guide decisions (appendix exhibit H.4). The program- wide participant ratings of 
usefulness rose from 3.69 in 2006-07 to 3.95 in 2007-08, to 3.96 in 2008-09, with 
the largest increases across years in ratings that gave credit to Center technical 
assistance for helping the participant’s organization to take action-maintaining or 
changing a policy or practice, taking next steps in a longer-term improvement 
effort, or solving a problem (appendix exhibit H.5). 


Exhibit 5.3. Mean ratings of technical quality, relevance, and usefulness, by 
center type and by year 



Technical quality 

Relevance 

Usefulness 


2006- 

07 

2007- 

OS 

2008- 

09 

2006- 

07 

2007- 

OS 

2008- 

09 

2006- 

07 

2007- 

08 

2008- 

09 

All Centers (N=21) 

3.34 

3.51 

3.57 

3.94 

4.08 

4.15 

3.69 

3.95 

3.96 

All RCCs (N=16) 

3.21 

3.41 

3.52 



m 

3.71 

3.99 

3.94 

All CCs (N=5) 

3.73 

3.86 

3.72 

3.78 


Hi 

3.65 

3.84 

4.01 

Difference of RCC 
and CC means 

-0.52 t 

-0.45 t 

-0.20 t 

0.21* 

0.22 t 

-0.02 

0.06 

0.15* 

-0.07 

Pooled standard 
deviation (all Centers) 

0.41 

0.41 

0.37 

0.34 

0.22 

0.24 

0.34 

0.23 

0.27 

Ratio of difference in 
means to pooled 
standard deviation 

- 1.28 

- 1.09 

- 0.55 

0.62 

1.00 

- 0.08 

0.18 

0.64 

- 0.26 


NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” rating is the mean of the 
ratings for the three quality dimensions. A notation of f indicates that the difference in the mean ratings between the 
CCs and RCCs within that year is at least one-half of one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.34 for the 2006-07 program year. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to project 
ratings; each project contributed equally to Center ratings; and each Center contributed equally to cross-Center 
ratings. 


computed as the standard deviation of the variable of interest (e.g., relevance) computed at the Center level. For 
analyses using the project as the unit of analysis, the pooled standard deviation was computed at the project level. 
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■ The RCCs’ average quality rating was higher in each successive year but 
remained lower than that of the CCs in each year. The average rating of the 
RCCs for the quality of sampled projects, rated by experts, was below that of the 
CCs (3.21 vs. 3.73 in 2006-07, 3.41 vs. 3.86 in 2007-08, and 3.52 vs. 3.72 in 
2008-09). The difference between RCC and CC means decreased from 0.52 in 
2006-07 to 0.20 in 2008-09 (exhibit 5.3). 

■ The average relevance rating for RCCs was higher than for CCs in the first two 
years, although CC ratings rose in each year. For 2006-07, the relevance rating 
(from project participants) averaged 3.99 for the RCCs and 3.78 for the CCs 
(difference of 0.21 between RCCs and CCs). In the following year the relevance 
ratings averaged 4.18 for RCCs and 3.96 for CCs (difference of 0.22 between 
types of Centers). In 2008-09, the mean was 4.15 for RCCs and 4.17 for CCs 
(exhibit 5.3). 

■ On the usefulness of sampled projects, there was no consistent difference between 
RCCs and CCs. The 2006-07 average rating for usefulness (from project 
participants) was similar across types of Centers (3.71 and 3.65). In 2007-08 the 
RCC average (3.99) was higher than that of CCs (3.84). In 2008-09 there was not a 
substantial difference between the averages for the two types of Centers (3.94 

and 4.01) (exhibit 5.3). 


Variation by Type of Project 

Analysis of the variation in ratings at the project level could provide information on the 
practices of Centers, indicating particular strengths of the Centers and areas for improvement. 
Perhaps the Centers were more successful, at least from the perspective of the reviewers, at some 
types of projects compared with others. Thus, the evaluation looked at different groups of 
projects across Centers that might be expected to show higher quality, relevance, or usefulness 
than other projects. 69 For example, projects that were larger in scope than others in the Center 
might have garnered higher ratings from the experts or the participants. Other potential sources 
of variation examined here included the types of activities or resources offered and the types of 
expertise brought to bear in delivering or reviewing the technical assistance. 

■ The distributions of ratings across projects were wider than the distributions 

across Centers, which on all three measures were clustered in the “ moderate to 
high ” range. Some projects were found in the “high to very high” range on each 
measure in each year (exhibit 5.4). There were also projects in each year below 
the “moderate” rating (i.e., in the “low to moderate” and the “low to very low” 
ranges combined) for quality and usefulness. For relevance, no sampled project 
was found below the “moderate” rating in 2007-08 or 2008-09. 


69 For analyses using the project as the unit of analysis, the pooled standard deviation was computed at the project 
level. 
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■ No sampled project received the lowest possible mean rating (“low to very low”) 

after 2006-07. For 2006-07 both RCCs and CCs had 4 percent of their projects 
rated “low to very low” for quality by experts; in the subsequent years they had 
none. For both relevance and usefulness, 1 percent of RCC projects had a “low to 
very low” rating from participants in 2006-07, but no project of any Center 
received this rating in 2007-08 or 2008-09 (exhibit 5.4). 


Exhibit 5.4. Percent of projects by rating for technical quality, relevance, and 
usefulness, by year 

Technical quality 


Percent of projects 



Very low to Low to Moderate High to very Very low to Low to Moderate High to very 

low moderate to high high low moderate to high high 

( 1 . 0 - 1 . 9 ) ( 2 . 0 - 2 . 9 ) ( 3 . 0 - 3 . 9 ) ( 4 . 0 - 5 . 0 ) ( 1 . 0 - 1 . 9 ) ( 2 . 0 - 2 . 9 ) ( 3 . 0 - 3 . 9 ) ( 4 . 0 - 5 . 0 ) 

RCCs CCs 
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Exhibit 5.4. (continued) 
Relevance 


Percent of projects 



Very low to Low to Moderate High to very Very low to Low to Moderate High to very 

low moderate to high high low moderate to high high 

(1.0-1. 9) (2.0-2.9) (3.0-3.9) (4.0-5.0) (1.0-1.9) (2. 0-2.9) (3.0-3.9) (4.0-5.0) 

RCCs CCs 

Usefulness 


Percent of projects 



Very low to 

Low to 

Moderate 

High to very 

Very low to 

Low to 

Moderate 

High to very 

low 

moderate 

to high 

high 

low 

moderate 

to high 

high 

(1.0-1. 9) 

(2.0-2.9) 

(3.0-3.9) 

(4. 0-5.0) 

(1.0-1. 9) 

(2. 0-2.9) 

(3.0-3.9) 

(4. 0-5.0) 


RCCs CCs 

EXHIBIT READS: For the 2006-07 program year, tour percent of the projects sampled from RCCs had a mean quality 
rating of very low to low (between 1 .0 and 1 .9 on a 5-point scale). Twenty-nine percent of the sampled RCC projects 
had a mean quality rating of low to moderate (2.0 to 2.9 on a 5-point scale) in that year. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to project 
ratings. 
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Ratings by Project Activities 

The technical assistance provided by the Centers varied in delivery methods, including the 
types of activities offered. The following types of activities and resources were found across the 
sampled projects: 70 

■ Ongoing consultation and follow-up 

■ Research collections and syntheses 

■ Engagement of participants in project planning 

■ Training events 

■ Task force meetings and work 

■ Conferences 

■ Support for development of a formal plan to implement a program or policy 

We examined whether projects that included certain types of technical assistance activities 
received more favorable project ratings either from experts knowledgeable about the topic or from 
participants in the project. By comparing the ratings of projects that offered each activity with 
ratings of projects that did not, strengths and weaknesses of particular modes of Center work might 
be discernible. However, no clear pattern of differences in ratings by activity type was found across 
all three years (appendix exhibits H.6-H.8). 


Ratings by Whether the Project Addressed Systems of Support 

Because technical assistance with statewide systems of support or school support teams 
was in high demand from the states (as discussed in chapter 4), projects addressing the topic of 
“effective systems of support” might have been rated higher than others with respect to relevance 
or usefulness. And because it was an area of considerable Center activity, the Centers’ depth of 
experience with the topic might have helped these projects gain relatively high ratings for 
quality. However, no differences were found (exhibit 5.5). 


70 For each sampled project, the Center furnished the team with a cover sheet intended to inform the expert 
reviewers about the project context, purpose, activities, and products. As described in chapter 2 and appendix C, the 
team coded the content of each project cover sheet to describe the activities and resources offered as part of the 
project. When a cover sheet indicated that the project had multiple components, multiple codes were assigned. 
Chapter 3 presents findings on the frequency with which each year’s sampled projects from RCCs and CCs 
respectively included each activity. 
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Exhibit 5.5. 


Technical quality, relevance, and usefulness ratings of SOS 
projects vs. all other projects in the study sample, by year 



Technical quality 

Relevance 

Usefulness 


2006- 

2007 - 

2008- 

2006- 

2007 - 

2008- 

2006- 

2007 - 

2008- 

Project Topic 

07 

08 

09 

07 

08 

09 

07 

08 

09 

SOS projects 

(2006-07 n=36, 

2007- 08 n=39, 

2008- 09 n=33) 

3.41 

3.51 

3.62 

4.00 

4.18 

4.19 

3.71 

3.93 

3.98 

All other projects 

(2006-07 n=87, 

2007- 08 n=82, 

2008- 09 n=86) 

3.32 

3.54 

3.58 

3.94 

4.12 

4.15 

3.69 

3.94 

3.97 

Difference in means 

0.09 

- 0.03 

0.04 

0.07 

0.06 

0.04 

0.02 

- 0.01 

0.01 

Pooled standard deviation 

0.69 

0.65 

0.61 

0.52 

0.43 

0.43 

0.54 

0.48 

0.48 

Ratio of difference in 
means to pooled standard 
deviation 

0.13 

- 0.05 

0.07 

0.13 

0.15 

0.09 

0.03 

- 0.02 

0.02 


f The difference in the mean ratings within that year of SOS projects and all other projects exceeds one-half of the 
pooled standard deviation in the rating. 

EXHIBIT READS: For the 2006-07 program year, the mean technical quality rating for SOS projects was 3.41 while 
the mean technical quality rating for all other projects in the study sample was 3.32. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 
rating, and each project contributed equally to the mean rating for its category. 


Ratings for RCC Projects by Whether They Drew on CC Contributions 

Cooperation between CCs and RCCs was one element of the design of the Center 
program, as discussed throughout this report. Thus, among the types of resources that RCC 
projects might offer to their participants were materials developed by CCs, activities developed 
with advice from CCs, or technical assistance from CC staff members. One hypothesis for the 
evaluation was that the RCC projects incorporating CC contributions would be rated higher for 
technical quality than RCC projects that were developed without CC input, since the CCs were 
specifically charged with synthesizing the knowledge base in their areas of focus, and since in 
each year the CCs had a higher mean rating of technical quality than the RCCs. 

■ Compared with other RCC projects, the ones that included CC contributions 

were rated higher for technical quality in one of three years studied. In 2008-09 
the sampled RCC projects that included one or more types of contributions from a 
CC were rated more than one-half of a standard deviation higher than those that 
did not (exhibit 5.6). No differences greater than one-half of a standard deviation 
were found for earlier years. 
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Exhibit 5.6. 


Technical quality, relevance, and usefulness ratings of RCC 
projects, by presence or absence of CC contributions, by year 



Technical quality 

Relevance 

Usefulness 

CC contribution or no 

2006- 

2007 - 

2008- 

2006- 

2007 - 

2008- 

2006- 

2007 - 

2008- 

CC contribution 

07 

08 

09 

07 

08 

09 

07 

08 

09 

CC contribution 

(2006-07 n=46, 

2007- 08 n=32, 

2008- 09 n=45) 

3.40 

3.59 

3.72 

3.92 

4.23 

4.13 

3.63 

3.99 

3.97 

No CC contribution 

(2006-07 n=50, 

2007- 08 n=63, 

2008- 09 n=48) 

3.09 

3.36 

3.39 

4.09 

4.18 

4.19 

3.78 

3.97 

3.94 

Difference in means 

0.31 

0.22 

0.33 1 

- 0.17 

0.05 

- 0.05 

- 0.15 

0.02 

0.02 

Pooled standard 
deviation 

0.67 

0.63 

0.61 

0.53 

0.42 

0.43 

0.56 

0.48 

0.47 

Ratio of difference in 
means to pooled 
standard deviation 

0.47 

0.35 

0.54 

- 0.32 

0.12 

- 0.12 

- 0.27 

0.05 

0.05 


f The difference in the mean ratings within that year between the RCC projects with and without CC contributions 
exceeds one-half of the pooled standard deviation in the rating. 

EXHIBIT READS: For the 2006-07 program year, the mean technical quality rating for RCC projects with a CC 
contribution was 3.40 while the mean technical quality rating for RCC projects with no CC contribution was 3.09. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 
rating, and each project contributed equally to the mean rating for its category. 


Ratings for CC Projects by Whether They Drew on RCC Contributions 

Also as a part of the cooperation that was expected within the overall Center program, 
some CC projects drew on the RCCs to help engage participants or to provide substantive help 
with technical assistance (as described in chapter 3). The evaluation team hypothesized that the 
CC projects that had RCC involvement might have been rated higher for relevance than other CC 
projects since the RCCs were charged with knowing the specific purposes and priorities of states 
in their regions, and since their projects received higher average ratings for relevance than CC 
projects in 2006-07 and 2007-08. However, the ratings of sampled projects did not show such a 
difference. 


■ CC projects with RCC contributions were not rated differently than other CC 
projects for relevance or usefulness; they were rated lower than other CC 
projects for quality in one year. Across years, participants’ ratings of CC projects 
with and without RCC contributions showed no differences greater than one-half 
of a standard deviation in relevance or usefulness (exhibit 5.7). CC projects with 
RCC contributions had quality ratings from experts that were more than one-half 
a standard deviation lower in 2007-08 but not in the years before or after. 
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Exhibit 5.7. 


Technical quality, relevance, and usefulness ratings of CC 
projects, by presence or absence of RCC contribution, by year 



Technical quality 

Relevance 

Usefulness 

RCC contribution or no 

2006- 

2007- 

2008- 

2006- 

2007- 

2008- 

2006- 

2007- 

2008- 

RCC contribution 

07 

08 

09 

07 

OS 

09 

07 

OS 

09 

RCC contribution 

(2006-07 n=1 7, 

2007- 08 n=10, 

2008- 09 n=11) 

3.84 

3.59 

3.76 

3.72 

4.01 

4.16 

3.58 

3.87 

4.00 

No RCC contribution 

(2006-07 n= 10, 

2007- 08 n=16, 

2008- 09 n=1 5) 

3.54 

4.06 

3.69 

3.88 

3.87 

4.18 

3.75 

3.73 

4.09 

Difference in means 

0.30 

-0.47 + 

0.07 

- 0.17 

0.15 

- 0.02 

- 0.17 

0.14 

- 0.09 

Pooled standard deviation 

0.62 

0.62 

0.44 

0.46 

0.40 

0.44 

0.47 

0.44 

0.50 

Ratio of difference in 
means to pooled standard 
deviation 

0.49 

- 0.77 

- 0.04 

- 0.36 

0.37 

- 0.04 

- 0.36 

0.32 

- 0.18 


f The difference in the mean ratings within that year between the CC projects with and without RCC contribution 
exceeds one-half of the pooled standard deviation in the rating. 

EXHIBIT READS: For the 2006-07 program year, the mean technical quality rating for CC projects with a RCC 
contribution was 3.84 while the mean technical quality rating for CC projects with no RCC contribution was 3.54. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 
rating, and each project contributed equally to the mean rating for its category. 


Ratings by Source of Quality Assurance 

For 2008-09, the evaluation gathered Centers’ reports on the sources that they used for 
quality assurance on each sampled project. Possible sources of expertise in quality assurance, as 
discussed in chapter 3, included several that were essentially “in-house”: Center staff, other staff 
in the grantee organization, and other staff in subgrantee organizations. The external sources 
included outside experts retained to review drafts, CCs (in the case of a CC project, this would 
be a Center other than the one conducting the project), and the U.S. Department of Education. It 
seemed possible that projects that had undergone external review would receive relatively higher 
ratings for quality from the expert panelists who rated projects for this evaluation. 

■ Projects that had been reviewed for quality assurance by outside experts or by a 
CC were rated higher for quality than other projects. The mean quality rating for 
projects reported to have been reviewed by experts was 3.73, compared with 3.42 
for other projects (exhibit 5.8). For projects reportedly reviewed by CCs, the 
mean was 3.83, compared with 3.46 for other projects. No differences greater than 
one-half of a standard deviation were found for quality assurance by other 
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sources, including sources closely associated with the Center conducting the 
project (i.e., Center staff or other staff of the grantee or subgrantee organization). 


Exhibit 5.8. Technical quality, relevance, and usefulness ratings by source of 
quality assurance, 2008-09 


Source of quality 
assurance 

Technical 

Quality 

Relevance 

Usefulness 

Outside experts (n=63) 

3.73 

4.12 

3.94 

No outside experts (n=55) 

3.42 

4.21 

4.02 

Difference in means 

0.31 f 

- 0.09 

- 0.08 

Pooled standard deviation 

0.61 

0.43 

0.47 

Ratio of difference in means 
to pooled standard deviation 

0.51 

- 0.21 

- 0.16 

Content Centers (n=39) 

3.83 

4.14 

3.92 

No Content Centers (n=79) 

3.46 

4.18 

4.01 

Difference in means 

0.37 t 

- 0.04 

- 0.08 

Pooled standard deviation 

0.61 

0.43 

0.47 

Ratio of difference in means 
to pooled standard deviation 

0.60 

- 0.09 

- 0.18 


f The difference in the mean ratings exceeds one-half of the pooled standard deviation in the rating. 

EXHIBIT READS: For the 2008-09 program year, the mean technical quality rating for projects that used 
outside experts was 3.73 while the mean technical quality rating for projects that did not use outside 
experts was 3.42. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project 
participants for relevance and usefulness. Responses weighted so that each panelist or participant 
contributed equally to the project rating, and each project contributed equally to the mean rating for its 
category. 


Relationships between the Relevance, Usefulness, and 
Quality of Center Projects 

In our analysis of the relevance, usefulness, and quality of Center technical assistance, we 
observed that some types of Centers, individual Centers, or groups of projects that received 
relatively high ratings for quality received relatively low ratings for relevance and usefulness, 
and vice versa. We hypothesized that differences in the rating and rankings (the rating relative to 
other groups of projects) could be due to the differences in the types of individuals who reviewed 
and scored projects for quality — content experts — and those who scored the relevance and 
usefulness — project participants. It was expected that these two groups might value and be better 
able to judge different qualities in a Center project, which is why we did not have content experts 
evaluate the projects for their utility or the participants assess the technical quality. 


85 



We examined the associations among the three dimensions more systematically by 
calculating correlation coefficients. These statistics indicate the strength and direction of a 
linear relationship between two factors. A correlation coefficient can vary from positive 1 .00 
(indicating a perfect positive relationship), through zero (indicating the absence of a 
relationship), to negative 1.00 (indicating a perfect negative relationship). If the correlation is 
statistically significant (p <.05), we can have strong (95 percent) confidence that the level of 
association we calculated is not due to chance. 

■ Participants’ ratings of relevance and usefulness were closely related 
(exhibit 5.9). The correlation coefficient was +0.84 for 2006-07, +0.79 for 2007- 
OS, and +0.83 for 2008-09. This indicates that the extent to which participants 
rated the projects as relevant was associated with how they deemed the project to 
be useful to their agency. These coefficients were all statistically significant at 
p<.05. 

■ There was no statistically significant relationship between ratings for the 
quality of Center projects and ratings for their relevance or usefulness 
(exhibit 5.9). The results indicated correlations ranging from -0.12 to +0.04 
between quality and relevance, and from -0.09 to +0.07 between quality and 
usefulness. Because the coefficients are not statistically significant we cannot be 
sure that they are different from zero (no relationship). Thus the correlations 
suggest that the expert rating of technical quality was not related to the extent to 
which participants deemed the projects to be relevant or useful. 


Exhibit 5.9. Correlations between project-level technical quality, relevance, 
and usefulness ratings, by year 


Spearman’s Rho 

Combination 

2006-07 

2007-08 

2008-09 

Quality and relevance 

- 0.12 

- 0.04 

+ 0.04 

Quality and usefulness 

- 0.04 

- 0.09 

+ 0.07 

Relevance and usefulness 

+ 0 . 84 * 

+ 0 . 79 * 

+ 0 . 83 * 


* The correlation is statistically significant at p<.05. 

EXHIBIT READS: For the 2006-07 program year, the correlation between project-level ratings of 
technical quality and relevance was a negative 0 . 12 . 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project 
participants for relevance and usefulness. Responses weighted so that each panelist or participant 
contributed equally to the project rating. 


71 For this analysis, the evaluation team used Spearman’s rank order correlation because this non-parametric rating is 
the appropriate statistical function to describe correlations between two variables where the values of the variables 
are not normally distributed and are on an ordinal, but not interval scale (such as ratings). 
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Variation by Participant Experiences and Roles 


The analyses by subgroups of projects, reported above, brought to light few types of 
projects that had consistently higher or lower ratings than other projects. However, it is possible 
that the targeting, tailoring, and engagement in technical assistance experienced by a particular 
participant would affect how he or she regarded it. In other words, the particular way in which 
the Centers delivered assistance to individuals — and the professional roles of the individuals on 
whom assistance focused — might account for differing perceptions of relevance and usefulness. 
If so, a project might have relevance and usefulness similar to those of other projects when the 
ratings are averaged across all participants in that project, but individual participants might give 
it systematically different ratings compared with other participants. 

Thus the team examined whether the rating of relevance or usefulness given by a 
participant was related to that participant’s experience in the project or to his or her job 
responsibilities. These analyses provided an opportunity to follow up on findings from the prior 
evaluation conducted on the Comprehensive Regional Assistance Centers established under the 
1994 ESEA reauthorization (described in chapter 1 of this report). " As some of the participant 
survey questions changed from the first to the second survey administration, examination of 
participant ratings based on those particular items are presented only for 2007-08 and 2008-09 
when the survey items remained the same. For survey questions that remained the same, all three 
years (2006-07, 2007-08, 2008-09) are presented. The specific participant characteristics 
examined included: 

■ Whether the participant contributed to the design of the project 

■ How much time the participant spent with project activities or resources 

■ The participant’s place of employment, and the extent to which the job focused on 
NCLB 

In the following discussion, the unit of analysis is the participant, not the project. The 
mean ratings are those provided by a group of participants, across all projects, who had a 
particular type of experience or background; within the same project there could be participants 
with different experiences and backgrounds. We first describe the variation found in each of 
these participant characteristics and the ways in which ratings did or did not differ according to 


72 U.S. Department of Education, Office of the Under Secretary, Planning and Evaluation Service, Elementary and 
Secondary Education Division. (2000). Comprehensive Regional Assistance Centers Program: Final Report on the 
Evaluation (Volume I). Washington, DC: Author. 

73 It is important to note that these participant-level analyses are exploratory, and the relationships described 
between participant characteristics and their ratings are correlational, not causal. The participant responses were 
weighted to represent all project participants in the sample frame for each project. The weights were calculated by 
dividing the total number of sample-eligible participants by the number of respondents completing a survey who 
participated in that project. The weighted figures shown in the tables in this section approach but never exceed the 
sample frame figures shown in appendix exhibit B.4, depending on the number of survey respondents who provided 
valid responses to the specific question! s) comprising each table. See chapter 2 and appendix B for more 
information on survey administration and the process used to weight responses. 
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each characteristic individually. The section concludes with a multivariate analysis examining 
each participant characteristic while controlling for the presence of the other characteristics. 


Ratings by Participants’ Involvement in Project Design 

An aspect of participation that was of particular interest, given the Centers’ charge to work 
closely with their clients, was the extent to which participants were involved in the projects’ design 
stages and, then, whether perceptions of relevance and usefulness might vary with a participant’s 
involvement. The evaluation of the earlier program of Comprehensive Regional Assistance 
Centers, based on survey results from a sample of participants, concluded: “Comprehensive Center 
technical assistance is more useful to customers if it is tailored to address their needs and 
interests.” 74 For the current evaluation, the team analyzed participant responses to a question 
asking respondents whether they were personally involved in determining the goals or designing 
the content or format of the project. 

■ A higher proportion of participants in the sampled RCC projects reported being 
involved in some aspect of determining project goals or designing the project, 
compared with participants in the CC projects. For 2007-08 the figures were 
39 percent of participants in RCC projects and 20 percent of those in CC projects; 
for 2008-09, 42 percent of participants in RCC projects and 16 percent of those in 
CC projects were involved either in determining goals or designing the projects 
(exhibit 5.10). In 2006-07, when this survey question included a 9-item list of 
ways in which a respondent might have been involved in determining goals or 
designing the project, more respondents indicated that they had been involved 
(57 percent for participants in RCC projects and 42 percent for those in CC 
projects), but the relatively higher rate of involvement in RCC projects was 
observed in that year as well (appendix exhibit H.9). In each of the three years, a 
chi square test found the difference across types of Centers to be statistically 
significant (pc.Ol). 75 


74 U.S. Department of Education, 2000, op cit. 

75 For the analysis of responses of project participants, inferential statistics were used to identify differences in 
responses among groups of participants that were greater than would be expected by chance alone. Because the 
sample of project participants were selected through a random sample of all project participants, the evaluators 
judged it appropriate to use test of statistical significance for these analyses. However, we would caution the reader 
not to generalize the results beyond the sample of projects selected for each year of the study, because the specific 
projects were selected on a non-probability basis. 
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Exhibit 5.10. Participants’ involvement in determining project goals or design, 
by type of Center and by year 


Percent of participants 


Involved in determining 
the goals or content of 
the project 

RCC projects 

CC projects 

2007-08 

(n=1,722) 

2008-09 

(n=1,701) 

2007-08 

(n=2,131) 

2008-09 

(n= 1,506) 

Total 

100% 

100% 

100% 

100% 

Yes 

39 

42 

20 

16 

No 

61 

58 

80 

84 


NOTE: Difference in the proportion of participants involved in design by type of Center is 
statistically significant (p<.05, Chi Square) in each year. 

EXHIBIT READS: for the 2007-08 program year, 39 percent of participants in projects 
conducted by RCCs reported they were involved in the design of the technical assistance. 

SOURCE: Surveys of project participants. Responses weighted so that respondents 
represented all participants for the project. 

■ Participants who reported being involved in determining the goals or designing 
the content of the project provided higher relevance and usefulness ratings in 
all years than did other participants. The average ratings in projects sampled for 
the current study corroborate the survey findings from the prior evaluation of the 
Centers. Participants who reported involvement in project design gave the 
projects a 4.06 average score for relevance and a 3.83 score for usefulness in 
2006-07 (exhibit 5.11). In contrast, average scores for participants not involved in 
project design were 3.81 and 3.56, respectively. These differences were 
statistically significant (with pc. 01 for differences in both relevance and 
usefulness). 

■ While the participants who were involved in planning gave higher ratings than 
those who were not, no difference in ratings was found between the projects 
that did or did not offer opportunities for involvement when analyzed at the 
project level. Comparing mean ratings at the project level, as reported earlier in 
this chapter, we did not find a difference greater than one-half of a standard 
deviation in relevance or usefulness ratings among projects that engaged 
participants in planning as opposed to those that did not (appendix exhibits H.6- 
H.8).This suggests that involving some participants in the design phase may make 
a difference in the ratings provided by those particular participants, but not in the 
ratings provided by all participants in the project. 
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Exhibit 5.11. Mean relevance and usefulness ratings, by respondent 

involvement in determining project goals or design, by year 


Involvement in project 
design 

Relevance 

Usefulness 

2006-07 

2007-08 

2008-09 

2006-07 

2007-08 

2008-09 

Involved in project design 

(2006-07 n=1, 802, 

2007- 08n=1,092, 

2008- 09 n=943) 

4.06 

4.27 

4.37 

3.84 

4.14 

4.20 

Not involved in project 
design 

(2006-07 n=1, 638, 

2007- 08 n =2,71 6, 

2008- 09 n=2, 134) 

3.81 

3.89 

3.90 

3.56 

3.77 

3.80 

Difference 

0.25* 

0.38* 

0.47* 

0.28* 

0.37* 

0.40* 

Significance 

pc.01 

p<.01 

p<.01 

pc.01 

pc.01 

pc.01 


* Difference statistically significant within each year at p<.05 using a one-way ANOVA. 


EXHIBIT READS: For the 2006-07 program year, participants who indicated that they were involved determining the 
goals or design of the project provided a relevance rating of 4.06 while project participants not involved in determining 
the goals or design of the project provided a relevance rating of 3.81 . 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all participants for 
the project. 
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Ratings by Time Spent on Project Activities 

The literature on professional development suggests that the duration of training is 
associated with better outcomes. 76 Similarly, the previous evaluation of the Comprehensive 
Centers program, based on survey data from participants, concluded: “Comprehensive Center 

77 

technical assistance is more useful to customers if it is intensive and if it extends over time.” 
Therefore, this study explored the possibility of a relationship between the time an individual 
spent in project activities and the ratings he or she provided, assessing whether the same 
relationship persisted. 

■ Participants who spent more time on project activities tended to give higher 
ratings of relevance and usefulness in 2007-08 and 2008-09. Among all 
participants, those who spent five days or less carrying out activities related to the 
project gave ratings for relevance and usefulness near the halfway point between 
“moderate” and “high”: for relevance, the mean rating given by this group was 
3.59 in 2007-08 and 3.58 in 2008-09; for usefulness the means were 3.46 and 3.42 
(exhibit 5.12). Those who spent more time (i.e., 6 or more days) gave mean 
ratings in the “high” range (near or above 4.0) that were significantly higher than 
ratings by participants who spent 5 days or less (pc.Ol for differences). For each 
type of rating in each year, the highest ratings were given by participants who had 
spent more than 10 days on the project. 78 


7h Desimone, L., Porter, A.C., Garet, M., Yoon, K.S., and Birman, B. (2002). “Does Professional Development 
Change Teachers' Instruction? Results from a Three-Year Study.” Educational Evaluation and Policy Analysis, 
24(2), 81-112; Garet, M., Porter, A., Desimone, L., Birman, B., and Yoon, K.S. (2001). “What Makes Professional 
Development Effective? Results from a National Sample of Teachers.” American Educational Research Journal, 
38(4), 915-945. 

77 U.S. Department of Education, 2000, op cit., p. 9. 

78 For 2006-07, respondents indicated the amount of time spent in each of eight activities by selecting from the 
following five options: “More than 5 days,” “3-5 days,” “1-2 days,” “Less than 1 day,” and “Not applicable.” Thus 
the findings are not directly comparable across years. However, a relationship between time spent and ratings of 
relevance and usefulness was found for that year as well. For each of eight project-related activities addressed in the 
survey for 2006-07, participants who reported spending at least three days in the activity rated the project higher on 
both relevance and usefulness than did participants who spent less time in that activity (appendix exhibit H.10). All 
of the differences were statistically significant at the p<.05 level using a one-way ANOVA. In addition, there was a 
statistically significant relationship between the number of activities in which the participant spent three or more 
days and the rating of project relevance. The correlation between the number of different project activities in which 
the respondent participated for three or more days and the rating of relevance was +0.41, while the correlation with 
the rating of usefulness was +0.34. These correlation coefficients were statistically significant at p<.05 
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Exhibit 5.12. Relevance and usefulness ratings, by amount of time participants 
spent on project activities, by year 



Relevance 

Usefulness 

Time Spent on Project 
Activities 

2007-08 

(n=3,820) 

2008-09 

(n =3,0 45) 

2007-08 

(n=3,791) 

2008-09 

(n=3,029) 

More than 30 days 

(2007-08 n=620, 
2008-09 n=400) 

4.46 

4.35 

4.28 

4.20 

21 to 30 days 

(2007-08 n=241 , 
2008-09 n=223) 

4.07 

4.50 

3.89 

4.32 

11 to 20 days 

(2007-08 n=690, 
2008-09 n=665) 

4.33 

4.35 

4.29 

4.24 

6 to 10 days 

(2007-08 n=840, 
2008-09 n=736) 

4.06 

4.13 

3.97 

4.05 

5 days or less 

(2007-08 n= 1,429, 
2008-09 n=1 ,021) 

3.59 

3.58 

3.46 

3.42 

Range (Difference 
between highest and 
lowest rating) 

0.87* 

0.92* 

0.83* 

0.90* 

Significance 

p<.01 

p<.01 

p<.01 

p<.01 


* Within each scale and each year, the difference in ratings by time spent on project activities is 
statistically significant (p<.05, ANOVA). 

EXHIBIT READS: For the 2007-08 program year, project participants involved in project activities 
for more than 30 days had an average relevance rating of 4.46. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented 
all participants for the project. 


Ratings by Participant Roles 

Were the sampled projects rated differently by participants who occupied different roles 
in their day-to-day jobs? The team investigated the possibility of a relationship between the 
participant’s place of employment or job responsibilities and the ratings he or she provided for a 
project’s relevance or usefulness. 

Given the intended emphasis of the Center technical assistance on building capacity at 
the state level, it is important to understand the extent to which the views of participants from 
state agencies might have differed from those of other participants. Of the sampled participants 
in RCC projects, the percent employed by SEAs was 41 percent in 2006-07, 54 percent in 
2007-08, and 56 percent in 2008-09 (exhibit 5.13). The percent employed by other agencies 
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varied across years. 79 In addition to intermediate agencies, local education agencies, and 
schools, other employers included institutions of higher education, nonprofits working with the 
state, and other RCCs. 


Exhibit 5.13. Where participants in RCC projects were employed, by year 


Employed by: 

Percent of participants 

2006-07 2007-08 2008-09 

(n=2,415) (n=1,603) (0=1,474) 

Total 

1 00% 

1 00% 

1 00% 

State education agency (SEA) 

41 

54 

56 

Intermediate education agency 

14 

21 

21 

Local education agency (LEA) 

15 

7 

10 

School 

20 

11 

7 

Other 

10 

7 

5 


EXHIBIT READS: Among the surveys completed by participants in RCC projects, 

41 percent were completed by respondents who were employed in a state education 
agency for the 2006-07 program year. 

SOURCE: Surveys of project participants. Responses weighted so that respondents 
represented all participants for the project. 


■ In both 2007-08 and 2008-09, participants employed in SEAs gave higher 
ratings for relevance and usefulness to RCC projects than participants 
employed in other agencies. The SEA employees gave relevance ratings 
averaging 4.13 in 2007-08 and 4.16 in 2008-09; other participants’ relevance 
ratings averaged 3.95 in each of those years (exhibit 5.14). For usefulness, the 
SEA employees’ ratings averaged 3.92 in 2007-08 and 4.01 in 2008-09, compared 
with 3.78 and 3.65 for other participants. These differences were statistically 
significant (p<.01) in 2007-08 and 2008-09 for both relevance and usefulness. 
SEA employees’ ratings of relevance and usefulness did not differ significantly 
from those of other participants in 2006-07. 


79 Participants who were employed by agencies other than the SEA were considered “state-level” participants 
because they carried out state-level responsibilities such as membership in a school support team or state-level task 
force (see chapter 2 for more discussion of the definition of “state -level” participants). For this analysis, however, 
we differentiate among participants by their place of employment. 


93 



Exhibit 5.14. Mean relevance and usefulness ratings for RCC projects, by 
agency where respondent worked, by year 


Agency where 
respondent worked 

Relevance 

Usefulness 

2006-07 

2007-08 

2008-09 

2006-07 

2007-08 

2008-09 

State education agency 
(SEA) 

(2006-07 n=977, 

2007- 08 n=909, 

2008- 09 n=826) 

3.99 

4.13 

4.16 

3.68 

3.92 

4.01 

All other agencies 

(2006-07 n=1, 409, 

2007- 08 n=850, 

2008- 09 n=632) 

3.94 

3.95 

3.95 

3.68 

3.78 

3.65 

Difference 

0.05 

0.19* 

0.21* 

0.00 

0.14* 

0.36* 

Significance 

p=.13 

pc.01 

p<.01 

p=. 93 

pc.01 

pc.01 


* Difference statistically significant within each year at p<.05 using a one-way ANOVA. 


EXHIBIT READS: For the 2006-07 program year, the mean relevance rating of RCC projects provided by 
participants in SEAs was 3.99 while the mean relevance rating by participants working at all other agencies 
was 3.94. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all 
participants for the project. 


In CC projects, most participants were employees of either an RCC or an SEA; no more 
than 2 percent were employed by any one other type of agency in any year (exhibit 5.15). The 
relative proportion of RCC and SEA staff participating in CC projects fluctuated, with more 
coming from RCCs in 2006-07 (59 percent RCC), more from SEAs in 2007-08 (54 percent), and 
similar proportions in 2008-09 (52 percent RCC, 48 percent SEA). 

■ hi both 2007-08 and 2008-09, CC projects were deemed more relevant and 
useful by RCC staff than by SEA staff RCC and SEA staff gave similar 
relevance and usefulness ratings to CC projects in 2006-07, but in each of the 
following years the RCC staff ratings rose while the SEA staff ratings changed 
little (exhibit 5.16). For 2008-09 RCC staff gave mean ratings of 4.17 and 4.14 for 
the relevance and usefulness of CC projects, compared with the 3.87 and 3.76 
ratings given by SEA staff. 
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Exhibit 5.15. Where participants in CC projects were employed, by year 



Percent of participants 


2006-07 

2007-08 

2008-09 

Employed by: 

(n=1 ,141) 

(n=2, 135) (n= 1,379) 

Total 

1 00% 

1 00% 

1 00% 

RCCs 

59 

41 

52 

State education agency (SEA) 

38 

54 

48 

Intermediate education agency 

k 

0 

k 

Local education agency (LEA) 

1 

k 

0 

School 

2 

2 

0 

Other 

k 

3 

0 


* Less than 0.5 percent. 

EXHIBIT READS: Among the surveys completed by participants in CC projects, 
59 percent were completed by respondents who were employed by RCCs for the 
2006-07 program year. 

SOURCE: Surveys of project participants. Responses weighted so that 
respondents represented all participants tor the project. 


Exhibit 5.16. Mean relevance and usefulness ratings for CC projects, RCC staff 
vs. state-level staff, by year 


RCC or state-level 

Relevance 

Usefulness 

staff 

2006-07 

2007-08 

2008-09 

2006-07 

2007-08 

2008-09 

RCC Staff 

(2006-07 n=661 , 

2007- 08 n=842, 

2008- 09 n=712) 

3.92 

4.07 

4.17 

3.81 

3.95 

4.14 

State-level staff 

(2006-07 n=484, 

2007- 08 n= 1 ,288, 

2008- 09 n=712) 

3.91 

3.89 

3.87 

3.73 

3.87 

3.76 

Difference 

-0.01 

0.18* 

0.30* 

0.08 

0.08* 

0.38* 

Significance 

CO 

CO 

II 

Q_ 

pc.01 

pc.01 

p=.1 2 

CO 

o 

N 

Q_ 

pc.01 


* Difference statistically significant within each year at p<.05 using a one-way ANOVA. 


EXHIBIT READS: For the 2006-07 program year, the mean relevance rating of CC projects provided by RCC staff 
was 3.92 while the mean relevance rating of CC projects by state-level staff was 3.91 . 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all participants for 
the project. 

Finally, because the Centers’ charge focused on assistance related to NCLB, the team 
hypothesized that participants whose regular job responsibilities focused most heavily on that 
law might give relatively higher ratings for relevance and usefulness of the sampled projects. 
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■ Participants who, in their regular jobs, spent more than 25 percent of their time 
on NCLB-related state responsibilities gave higher ratings to Center projects 
than participants whose jobs focused less on NCLB. We found that those who 
spent more than one quarter of their time at work on NCLB gave significantly 
higher relevance and usefulness ratings, compared with those who spent 25 
percent of their time or less on NCLB (pc.Ol for differences in both relevance and 
usefulness using ANOVA) (exhibit 5. 17). 80 

Exhibit 5.17. Mean relevance and usefulness ratings, by percent of time 
respondent spent on NCLB-related tasks in his/ her job and 
by year 


Percent of time spent on 
NCLB-related tasks 


Relevance 


Usefulness 

2006-07 

2007-08 

2008-09 

2006-07 

2007-08 

2008-09 

More than 75 percent 

(2006-07 n=908, 

2007- 08 n=738, 

2008- 09 n=924) 

4.05 

3.97 

4.17 

3.87 

3.84 

4.02 

51 to 75 percent 

(2006-07 n=541, 

2007- 08 n=613, 

2008- 09 n=587) 

3.98 

4.20 

4.13 

3.78 

4.07 

4.12 

26 to 50 percent 

(2006-07 n=493, 

2007- 08 n=366, 

2008- 09 n=549) 

3.94 

3.92 

4.26 

3.72 

3.88 

4.00 

25 percent or less 

(2006-07 n=476, 

2007- 08 n=478, 

2008- 09 n=763) 

3.72 

3.62 

3.74 

3.47 

3.75 

3.59 

Range (Difference between 
highest and lowest rating) 

Significance 

0.33* 

pc.01 

0.58* 

pc.01 

0.51* 

pc.01 

0.40* 

pc.01 

0.32* 

pc.01 

■ 


* Difference statistically significant within each year at p<.05 using a one-way ANOVA. 


NOTE: The survey question for this item differed slightly from the 2006-07 administration to the form used for 2007- 
OS and 2008-09 in that in the first administration, respondents were first asked to describe the specific aspects of 
NCLB implementation that were included in their job responsibilities and were then asked to describe the percent of 
time they spent on all of those tasks. For the subsequent administrations of the survey respondents were asked just 
to describe the percent of their time was spent on responsibilities related to NCLB. 

EXHIBIT READS: Among project participants for the 2006-07 program year, those whose job responsibilities included 
NCLB-related tasks more than 75 percent of the time provided a relevance rating of 4.05 while participants whose job 
responsibilities included NCLB-related tasks from 51 to 75 percent of the time provided a relevance rating of 3.98. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all participants for 
the project. 


80 The form of the question changed across years. For 2006-07, participants were asked in the survey to indicate 
what percentage of their hours on the job, in total, they spent working on any of the specific state-level 
responsibilities related to NCLB. In the subsequent years they were simply asked how much time they spent 
working on responsibilities related to NCLB. In each year they were given a choice of four percentage ranges: 0-25 
percent, 26-50 percent, 51-75 percent, and 76-100 percent. 
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Because participant characteristics may be interrelated, we conducted multivariate 
analyses to examine the associations of participant characteristics with project ratings of 
relevance and usefulness when holding the other participant characteristics constant. Multiple 
regressions of the four participant characteristics on ratings of project relevance and on ratings of 
project usefulness were consistent with analyses of individual participant characteristics. Each of 
the project characteristics was shown to have a statistically significant relationship with ratings 
of relevance and ratings of usefulness after controlling for the other three predictors (see 
appendix exhibits H.lland H.12). 
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Appendix A 
Center Lead Grantee 
and Subgrantee Organizations 




Center Lead Grantee 
and Subgrantee Organizations 


Center name 

Lead grantee and subgrantees 

Alaska Comprehensive 
Center 

■ 

Southeast Regional Resource Center (SERRC of Alaska) 
o Northwest Regional Educational Laboratory (NWREL) 

Appalachia Regional 
Comprehensive Center 

■ 

Edvantia 

o George Washington University - Center for Equity and Excellence 
in Education (CEEE) 

o Eastern Stream Center on Resources and Training (ESCORT) 
o National Center for Family Literacy (NCFL) 
o University of North Carolina at Greensboro - SERVE Center 
o Southern Regional Education Board (SREB) 

California 

Comprehensive Center 

■ 

WestEd 

o American Institutes for Research (AIR) 
o School Services of California 

Florida and Islands 
Comprehensive Center 

■ 

Educational Testing Service (ETS) 
o University of South Florida - David C. Anchin Center 
o Edvantia 

o Eastern Stream Center on Resources and Training (ESCORT) 
o Florida Association of School Administrators (FASA) 
o International Center for Leadership in Education (ICLE) 
o JLM Professional Education Services 
o Robin Wheeler, LLC 

Great Lakes East 
Comprehensive Center 

■ 

Learning Point Associates 
o RMC Research Corporation 
o American Institutes for Research (AIR) 
o Center for Applied Linguistics (CAL) 
o University of Michigan - Consortium for Policy Research in 
Education (CPRE) 

Great Lakes West 
Comprehensive Center 

■ 

Learning Point Associates 
o American Institutes for Research (AIR) 
o Mid-continent Research for Education and Learning (McREL) 
o University of Wisconsin - Wisconsin Center for Educational 
Research (WCER) 

o University of Michigan - Consortium for Policy Research in 
Education (CPRE) 

Mid-Atlantic 
Comprehensive Center 

■ 

George Washington University - Center for Equity and Excellence in 
Education (CEEE) 
o Edvantia 

o Eastern Stream Center on Resources and Training (ESCORT) 
o Group Jazz 

o Southern Regional Education Board (SREB) 

Mid-Continent 
Comprehensive Center 

■ 

University of Oklahoma 
o Northrop Grumman Information Tech 
o Accion Social Comunitaria 

o Mountain Plains Regional Resource Center (MPRRC) 
o North Central Regional Resource Center (NCRRC) 
o Southeast Regional Resource Center (SERRC of Alabama) 

New England 
Comprehensive Center 

■ 

RMC Research Corporation 
o Education Development Center (EDC) 
o WestEd 

o Education Alliance at Brown University 
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Center Lead Grantee 

and Subgrantee Organizations (continued) 


Center name Lead grantee and subgrantees 


New York 

Comprehensive Center 

■ 

RMC Research Corporation 
o Education Development Center (EDC) 
o WestEd 

o Education Alliance at Brown University 
o United Federation of Teachers Teacher Center (UFTTC) 

North Central 
Comprehensive Center 

■ 

Mid-continent Research for Education and Learning (McREL) 

Northwest Regional 
Comprehensive Center 

■ 

Northwest Regional Educational Laboratory (NWREL) 
o RMC Research Corporation 

Pacific Comprehensive 
Center 

■ 

Pacific Resources for Education and Learning (PREL) 

Southeast 

Comprehensive Center 

■ 

Southwest Educational Development Laboratory (SEDL) 
o Center for the Education and Study of Diverse Populations 
(CESDP) 

Southwest 

Comprehensive Center 

■ 

WestED 

o RMC Research Corporation 
o American Institutes for Research (AIR) 

Texas Comprehensive 
Center 

■ 

Southwest Educational Development Laboratory (SEDL) 
o Center for the Education and Study of Diverse Populations 
(CESDP) 

Assessment and 
Accountability Content 
Center 

■ 

WestEd 

o National Center for Research on Evaluation Standards, and Student 
Testing (CRESST) 

Center for Innovation 
and Improvement 

■ 

Academic Development Institute (ADI) 

o Temple University - Center for Research in Human Development 
and Education (CRHDE) 
o Little Planet Learning 

Center on Instruction 

■ 

RMC Research Corporation 
o Florida Center for Reading Research (FCRR) 
o RG Research Group 
o Horizon Research 

o Texas Institute for Measurement, Evaluation and Statistics (TIMES) 
o University of Texas at Austin - Vaughn Gross Center for Reading 
and Language Arts 

National Comprehensive 
Center on Teacher 
Quality 

■ 

Learning Point Associates 
o Education Commission of the States (ECS) 
o Educational Testing Service (ETS) 
o Vanderbilt University 

National High School 
Center 

■ 

American Institutes for Research (AIR) 
o Learning Point Associates 
o MDRC 

o National Center for Educational Accountability (NCEA) 
o WestEd 


EXHIBIT READS: The Alaska Comprehensive Center was operated by the lead grantee organization named the 
Southeast Regional Resource Center (of Alaska) with support from the subgrantee organization named the 
Northwest Regional Educational Laboratory. 


Source: U.S. Department of Education; individual RCC and CC websites 
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Appendix B 
Study Sample: 

Center Projects and Project Participants 




Study Sample 


This appendix section presents the Project Inventory Forms (PIFs) that were collected to 
identify the sample of projects to be included in the study as well as the documents sent to 
Centers requesting materials for expert panel review of sampled projects. The appendix 
concludes with historical tables showing the number of projects in the study sample for 2006-07 
and 2007-08 by topic and detailed information describing the administration of the participant 
surveys across all three years. 


Project Inventory Form 

The PIF was developed by the study team as a standard way for Centers to provide an 
inventory of their work for each program year. When asked to complete the inventory forms, 
Center Directors were informed about the specific project year for which information was being 
gathered as well as the purpose of this information to be used in the selection of a sample of 
projects for review of quality by expert panels to be conducted for the national evaluation. When 
completing the PIF, each Center was asked to nominate several projects that they believed best 
represented the work undertaken by that Center. Additional projects were purposefully sampled 
from the remaining projects on each center’s inventory to reflect each center’s overall portfolio 
of work, as well as the work of all regional or content centers in key topic areas. A sample PIF 
was provided including examples of the kinds of projects that should be listed by the centers and 
examples of activities and resources defining each project level of effort - major, moderate, 
minor. Examples of some projects, activities, and deliverables that should not be included on the 
inventory form were also provided, including the following: 

■ Training or professional development for Center staff 

■ Work on coordinating committees within the Centers 

■ Annual needs assessment activity or negotiations with states, unrelated to specific 
projects 

■ Other internal working meetings or documents 

When completing the PIF, each center provided a list of each project under the 
appropriate topic heading. While the Centers submitted PIFs using 14 topic areas for the 2006-07 
data collection, the evaluation team used the coding process described in appendix C to recode 
the substantive knowledge base of the project into a more detailed set of 22 topics. For the 2007- 
OS and 2008-09 PIF submissions, the Centers assigned one of the 22 topic areas to each project 
themselves as part of the submission. 

This appendix includes copies of the materials sent to the Centers to help them complete 
their PIFs including the instructions for completion (page B-6), the list of topic areas used for the 
2007-08 and 2008-09 data collections (page B-14), and a sample section of a PIF (page B- 16). 
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Letters Requesting Materials for Expert Panel Review from Centers 

For each data collection cycle, each Center received a notification letter from the 
evaluation team that listed the projects from their PIF that had been selected for expert panel 
review followed by guidance in assembling the materials for panel review, compiling lists of 
project participants for the participant survey, and completion of the cover sheet for each sampled 
project. The initial request for materials letter and supporting information starts on page B-22. The 
evaluation team reviewed the materials submitted by the Centers for accuracy and completeness 
and in cases where there were missing or incomplete materials or where further explanation was 
needed, followed up with Centers using the follow-up memo provided later in this appendix 
starting on page B-29. 


Decision rules for sampling projects from the PIFs 

Exhibit B.l below shows the decision rules the evaluation team applied in selecting the 
sample of projects for the evaluation sample. This six-step process was used to draw the 
evaluation in each of the three evaluation cycles. 


Exhibit B.l. Decision rules for sampling projects within each center 


Step 1. Determine the total number of projects to be sampled for that center 

Step 2. Identify the designated number of center-nominated projects, selected with certainty 

Step 3. Determine the number of additional projects to be sampled 

Step 4. Sample Major projects 

■ If the number of major projects was less than or equal to number of additional 
projects needed, sample all major projects and go to step 5 

■ If the number of major projects is greater than the number of additional projects 
needed, randomly select projects while controlling for topic and state (i.e., do not 
select multiple projects from any given topic or state [for centers serving multiple 
states] until all topics and states have at least one project sampled) - repeat until 
the sample size is met 

Step 5 Determine the number of additional projects to be sampled (if any) 

Step 6. Sample Moderate projects 

■ If the number of moderate projects is less than or equal to number of additional 
projects needed, sample all moderate projects 

■ If the number of moderate projects is greater than the number of additional projects 
needed, randomly select projects while controlling for topic and state (i.e., do not 
select multiple projects from any given topic or state [for centers serving multiple 
states] until all topics and topics have at least one project sampled) - repeat until 
sample size is met 
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Distribution of projects and projects in the evaluation sample 


Exhibits B.2 and B.3 included in this appendix (starting on page B-30), show the 
distribution of all major and moderate projects on the project inventory forms and projects in the 
sample by topic for the 2006-07 and 2007-08 data collection cycles. These tables are parallel to a 
similar table in the report for 2008-09 (exhibit 2.5). 


Administration of the Participant Survey 

As described in Chapter 2, the participant survey was administered to project participants. 
This included state-level staff for RCC projects and a combination of state-level and RCC staff 
for CC projects. Exhibit B.4 on page B-32 shows the number of participants at each stage of the 
data collection. Sampled respondents who indicated in their response to the first survey item that 
they did not participate in the project were excluded from analysis. 

For each of the three data collection cycles, the evaluation team analyzed the properties 
of the items included in the participant survey for constructing indices of relevance and 
usefulness. 

Principal components analysis with no rotation was conducted on each set of items in the 
relevance and usefulness scales to determine the underlying dimensions represented by these 
items. For each scale, analyses of each year of data resulted in a one-factor solution comprised of 
the same items and with comparable levels of reliability. For the relevance scale, the Cronbach 
alpha was 0.93 in 2006-07 and 0.94 in 2007-08 and 2008-09, and 0.95 for the usefulness scale in 
all three years. Additional information on the factor loadings and psychometric properties for the 
three years of data collection is provided in exhibits B.5 and B.6 starting on page B-33. 


Units of analysis and weighting for quality, relevance, and usefulness data 

The quality, relevance, and usefulness ratings of the sampled projects were presented in 
chapter 5 using several different units of analysis: the projects sampled from the program as a 
whole (all 21 Centers), the projects sampled for the 16 RCCs and the 5 CCs respectively, 
subgroups of the projects, and (for relevance and usefulness) subgroups of the participants. 

■ Program-wide and by Center type. The team weighted the mean center-level 
quality, relevance, or usefulness results equally so that each Center contributed 
equally to the mean for the entire program or for a set of Centers. 

■ Center-level. For a Center’s mean rating on quality, relevance, or usefulness, the 
team weighted each sampled project from that Center equally. 

■ Project-level. Several analyses were conducted for subgroups of projects across 
Centers — e.g., those projects that included a research synthesis among their 
products, or included training among their services. Each rater (an expert or 
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survey respondent) had equal weight in the individual project’s rating, and each 
project had equal weight in the subgroup mean rating. 

■ Participant-level. For some analyses of subgroups of participants across projects, 
the unit was the participant, not the project, and the weighting was designed to 
permit generalization to all the participants across all the sampled projects. Thus 
the weight for each participant’s response was the inverse proportion of the 
number of respondents compared with the total number of participants in the 
population for that project. The n’s shown in the participant-level tables in chapter 
5 reflect the weighted figures. The maximum number is the total number in the 
sample frame shown in exhibit B.4 in this appendix but the actual weighted n’s 
are often somewhat lower because the figures in the table represent a subset of 
participants or because some participants failed to respond to that survey item. 
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Project Inventory Form 


Center Project Inventory Form 


Dear Center Director, 

Enclosed are a Project Inventory Form and instructions to be used to compile a complete 
list of your Center’s projects during the current project year (July 1, 200X-June 30, 200X). 

The national evaluation will use the completed project inventories to select a sample of 
projects for review by expert panels. Please designate several projects that you believe best 
represent the work of your Center; these projects will be included among the sample of projects 
to be reviewed by expert panels. These panels will rate nominated and sampled projects for their 
quality. 

Please complete a draft inventory to review with a member of the national evaluation 
team during the site visit for the evaluation, scheduled to take place between April and June 
200X when [EVALUATION STAFF MEMBER] visits your Center. A final version of the 
project inventory should be completed as soon as possible after the site visit, but no later than 
June 30, 200X. Please return the completed inventory form to your evaluation liaison, [name] at 
@_ . 

If you have any questions about the inventory or the instructions for completing it, please 
contact at 1-xxx-xxxx or by email at @_ . 

Thank you for your continued support of the national evaluation of the comprehensive 
center program. 


According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless it displays a valid OMB control number. The valid OMB control number for this data collection 
instrument is 1850-8023. The time required to complete these worksheets is estimated to average 16 hours per 
response, including the time to review instructions, search existing data sources, gather the data needed, and fill in 
the form. Send comments regarding this burden estimate or any other aspect of this collection of information, 
including suggestions for reducing the burden, to the U.S. Department of Education, Washington, DC 20202-4651. 
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Project Inventory Form 


Instructions for Completing an Inventory of Projects 

Each Comprehensive Center will prepare an inventory of all of the projects that were 
active during the current grant period. The first inventory will cover the period from July 1, 2006 
through June 30, 2007. The second inventory will cover materials from July 1, 2007 through 
June 30, 2008, and the third inventory will cover July 1, 2008 through June 30, 2009. 

Projects that began in the current project year but will not be complete by June 30 should 
be entered in the inventory and noted as “ongoing.” 

Purpose 

This inventory will serve as the sampling frame for the expert panel reviews to be 
conducted for the national evaluation. A sample of projects will be selected in two ways. Each 
Center will nominate several projects that they believe best represents the work undertaken by 
that Center. The remaining projects will be purposefully sampled from the remaining projects on 
each Center’s inventory. Samples will be drawn to reflect each Center’s overall portfolio of 
work, as well as the work of all Regional or Content Centers in key topic areas. Items selected 
for review will be rated by an expert panel for their quality. 

If a project is nominated or sampled for expert panel review and rating, the evaluation 
team will ask your Center to collect and transmit all of the documents and other artifacts 
associated with that project (meeting agendas, briefing books, meeting summaries, training 
materials, white papers, web resources, etc.) for the expert panel to review. 

Identifying Projects for Inclusion in the Inventory 

The unit of analysis for the expert review panels will be the project. For the purposes of 
this inventory, a “project” will ordinarily comprise a group of closely related activities and/or 
deliverables designed to achieve a specific outcome for a specific audience. The Content 
Centers have some projects that consist of the development of a single product or deliverable, 
rather than a group of deliverables, but a project that comprises only a single product or 
deliverable will be the exception rather than the rule. The inventory should include all of the 
projects developed or delivered by the Center, including those developed or delivered in 
collaboration with other Centers, during the reporting period. 

Because each project listed in the inventory could potentially be sampled for expert panel 
review, each project shown as an entry (or row) in the inventory form should be a relatively 
complete project that can be understood and rated on its own by expert panel members who may 
not know anything about other aspects of the Center’s work. Although a single project may 
include a number of deliverables and activities, it will be designed to achieve a specific outcome 
and address (in almost all cases) a single topic. Where a group of activities and deliverables can 
be divided up into separate projects, each constituting a complete and coherent whole, the Center 
should list these as separate projects in the inventory. 

The following criteria should guide the Centers as they identify projects (and their 
associated activities and deliverables) for recording in the inventory form. They have been 
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Project Inventory Form 


developed to ensure that projects constitute units that are large enough for review and rating, but 
focused enough for coherence. Each project entered in the inventory should satisfy all three of 
the following criteria: 


■ Complete and coherent whole. Each project listed in the inventory should be 
able to stand on its own in an expert panel review. Avoid listing activities and 
deliverables as separate projects in the inventory if they cannot be understood or 
evaluated without information about related activities and deliverables. For 
example, a training event may require extensive planning (e.g., needs assessment, 
materials development) and follow-up activities (e.g., evaluation of the training, 
consultation on action plans). These planning and follow-up activities would be 
very difficult for a panel to rate in the absence of information about the event 
itself. Therefore the Center should list these planning and follow-up activities and 
the event itself as a single project (one row) on the inventory form. Each phase of 
the project — planning, the event itself, and follow-up — will be described briefly 
in the “activities and deliverables” column. Similarly, ongoing work with a state- 
level task force should be listed as a single project rather than each meeting of the 
task force being listed as a project. 

■ Common intended outcome. Where a cluster of activities and deliverables is 
designed by the Center to lead to the same outcome for the same audience(s), 
those activities and deliverables should be grouped as one project in the inventory 
form. On the other hand, where a set of activities is intended to produce more than 
one distinct outcome — for example, helping the state to develop a strategic plan 
for improving reading instruction, and helping the same state recruit and train 
literacy coaches — those two activities should be listed as separate projects. Where 
the Center replicates the same set of activities in each of several different states, 
that set of activities should be listed as a single project if the intended outcomes 
and processes do not differ materially from one state to another. Where the 
intended outcomes do differ substantially from state to state, the work in each 
state should be listed as a separate project. 

■ Topic area focus. With few exceptions, a project addresses just one of the 14 topic 
areas described in Exhibit A at the end of this document (e.g., state systems of 
support, reading/language arts). Where it is possible to divide a group of related 
activities into two different projects according to the topic area addressed, Centers 
should do so. For example, a regional forum on interventions for low-performing 
students in reading and mathematics that offers separate strands of sessions in each 
subject area should be listed as two different projects, one under reading/language 
arts and one under mathematics. In this case, all of the sessions on reading 
interventions would stand on their own as a complete and coherent whole for rating 
by an expert panel and should be listed as a separate project; the same would be true 
for the sessions on mathematics. (If a project cuts across topic areas and the 
activities and deliverables which it comprises cannot be divided up by topic area 
into complete and coherent units that would make sense to a review panel, the 


B-7 



Project Inventory Form 


project should be listed under the most relevant topic area with a note cross- 
referencing other topics, following the instructions for column IV below.) 

The sample inventory form at the end of this packet includes examples of the kinds of 
projects that should be listed by the Centers. 


Some projects, activities, and deliverables should not be included on the inventory form 
at all. These include: 


■ Training or professional development for Comprehensive Center staff 

■ Work on coordinating committees within the Comprehensive Center network 

■ Annual needs assessment activity or negotiations with states, unrelated to specific 
projects 

■ Other internal working meetings or documents 


Completing the Inventory Form 

Centers should use the attached form to complete their inventory of projects. Sample 
projects and examples of entries can be found in Exhibit B. 

Centers might find it useful to review their annual project plans, technical assistance 
plans, management plans or technical assistance logs as a starting point since those documents 
typically provide an overview of the various projects and activities that were planned for or 
conducted during the year. 

Once the inventory form is complete, Centers should designate which projects they would 
like to nominate for inclusion in the sample of projects reviewed by the expert panels, by inserting 
** after the name of the project in Column II. Centers should designate several projects. 

List each project under the appropriate topic heading. A list of topic headings, with 
definitions, is attached at the end of this document in Exhibit A. 

Directions for completing each column are as follows: 
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Topic Area Headings 

I. Project number 

II. Name 

III. Description 

IV. Additional topics 
addressed? 

V. Activities and 
Deliverables 


VI. Start Date 

VII. End Date 


List items under the appropriate topic area heading. Where a project 
fits under more than one topic heading, list it once, under the topic 
heading that is most relevant. Note the project’s relevance to other 
topics in Column IV. Add rows to the table as needed. Leave rows 
blank if your Center does no work in a particular topic area. 

Assign consecutive numbers to each item listed in the inventory. 
(Centers may want to complete this column as a final step, after all of 
the items have been entered in the inventory.) 

Enter the name of the project. Projects nominated by Center staff for 
review should be followed by ** in this column. 

Provide a concise description of the project. (See examples for 
appropriate level of detail.) 

If the project addresses more than one topic area, note that here. Use 
the list of topic areas provided in the appendix (this list corresponds 
with the row headings in the inventory). Entering the appropriate 
number from the list will save space. 

List all of the activities and deliverables associated with the project. 
(For products, include exact title, if applicable. For services, include 
location and type of participant.) Specific activities and deliverables 
may include: 

Meetings/conferences (includes items such as workshops, 
conferences, institutes, forums, webinars) 

Expert consultation/technical assistance (includes items such as 
assistance completing reports or applications, review of state plans, 
needs assessments, audits) 

Facilitation/support of working groups or teams (includes items 
such as planning meetings, participation in meetings, drafting 
summary documents) 

Guidance/information resources (includes items such as policy or 
issue briefs, fact sheets, congressional testimony, resource guides, 
planning tools, field guides, benchmarking rubrics, handbooks, 
exemplars, literature reviews/summaries, annotated bibliographies, 
case studies, websites) 

Training (includes items such as professional development 
materials/services, software, training materials) 

Other, specify 

Enter the start date for the project, including month and year. 


Enter the end date for the project, including month and year. 
If the project is currently ongoing, enter the note “ongoing.” 
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VIII: Major, 
Moderate or Minor 
Project 

IX: Target state(s), 
region(s), or regional 
center (s) 


X: Collaborations and 
Sources: Other CCs 


XI: Collaborations 
and Sources: Other 
TA Providers 


Indicate whether you consider this a major, moderate or minor project 
in terms of the relative level of effort and/or resources devoted to it. 


Regional Comprehensive Centers serving multiple states should note 
which state(s) participated in each project. Regional Centers serving a 
single state (Alaska, California, New York, and Texas) should note 
which region(s) within the state participated (e.g.. New York City vs. 
rest of state). Content Centers should specify which Regional Centers 
participated in the project. 

If the Center used materials developed by one of the Content Centers 
in the course of designing or delivering its own services, list this 
Content Center as a source. 

If the Center collaborated with another Regional Center or Content 
Center on the design, development and/or delivery of products and 
services, list this Center as a collaborator. 

Note collaborations with other Comprehensive Centers only in this 
column. 

Note collaborations with other technical assistance providers, such as 
Regional Educational Laboratories or universities, in this column. 

If the Center used materials developed by another TA provider in the 
course of designing or delivering its own services, list this provider as 

a source. 

If the Center collaborated with another TA provider on the design, 
development and/or delivery of products and services, list this 
provider as a collaborator. 
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National Evaluation of the Comprehensive Technical Assistance Centers 

Inventory of Projects 


Center Name: 


I. No. 

II. Name 

III. Description 

IV. Additional 
topics addressed? 

(From list in 
Exhibit A — enter 
appropriate 
number) 

V. Activities and 
Deliverables 

VI. Start Date 

VII. End 
Date (enter 
“ongoing” 
if project is 
not 

complete) 

VIII. 
Major, 
moderate 
or minor 
project 

IX. 

Target 

state(s), 

region(s) 

within a 

state, or 

regional 

centers(s) 

X. 

Collab- 

orations 

and 

Sources: 

Other 

CC’s 

XI. 

Collab- 

orations 

and 

Sources: 
Other TA 
Providers 

1. Components of Effective Systems of Support (State, District, School) 























2. Data Use / Data-driven Decision Making 























3. Formative Assessment 























4. Reading 























5. Adolescent Literacy 























6. Mathematics 























7. Drop Out Prevention 























8. High School Redesign/Reform 























9. Transition to High School 
























Project Inventory Form 
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I. No. 

II. Name 

III. Description 

IV. Additional 
topics addressed? 

(From list in 
Exhibit A — enter 
appropriate 
number) 

V. Activities and 
Deliverables 

VI. Start Date 

VII. End 
Date (enter 
“ongoing” 
if project is 
not 

complete) 

VIII. 
Major, 
moderate 
or minor 
project 

IX. 

Target 

state(s), 

region(s) 

within a 

state, or 

regional 

centers(s) 

X. 

Collab- 

orations 

and 

Sources: 

Other 

CC’s 

XI. 

Collab- 

orations 

and 

Sources: 
Other TA 
Providers 

10. Special Education - curriculum, instruction, and professional development 























11. Special Education - assessment 























12. English Language learners 























13. Highly Qualified Teacher Provisions of NCLB 























14. Teacher Preparation and Induction 























15. Teacher Professional Development 























16. Supplemental Education Services (SES) 























17. Response to Intervention (Rtl) 























18. Migrant Education 























19. Indian / Native American Education 























20. Data 

Management Compliance 
























Project Inventory Form 
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I. No. 

II. Name 

III. Description 

IV. Additional 
topics addressed? 

(From list in 
Exhibit A — enter 
appropriate 
number) 

V. Activities and 
Deliverables 

VI. Start Date 

VII. End 
Date (enter 
“ongoing” 
if project is 
not 

complete) 

VIII. 
Major, 
moderate 
or minor 
project 

IX. 

Target 

state(s), 

region(s) 

within a 

state, or 

regional 

centers(s) 

X. 

Collab- 

orations 

and 

Sources: 

Other 

CC’s 

XI. 

Collab- 

orations 

and 

Sources: 
Other TA 
Providers 

21. Assessment Design 























22. Parent Involvement 























23. Other 
























**Projects nominated by the Center for review by expert panels 


Project Inventory Form 




Project Inventory Form 


Exhibit A 

Topic Areas and Definitions 


Topic 

Definition 

1. Components of effective 
systems of support-state, 
district, school. 

This category included: (1) Design of state systems to meet NCLB 
requirements for statewide systems of support for districts and schools in 
corrective action or identified as in need of improvement [Sec. 1 1 17(a)]; (2) 
Interventions for schools or districts in need of improvement or corrective 
action; (3) For schools. Centers' projects addressed a range of options for 
school leadership and organization, professional development and classroom 
practice, and development and implementation of school improvement plans. 
For districts. Centers' projects typically addressed district leadership and 
development as well as implementation of district improvement plans. 

2. Data use/Data-driven 
decisionmaking. 

Use of data for educational program improvement. 

3. Formative assessment. 

Guidelines for implementing formative or benchmark assessment, 
comprehensive formative assessment systems. 

4 . Reading. 

Alignment of curriculum, instruction, and assessments with state standards in 
reading; teacher professional development in reading (not specific to 
adolescent literacy). 

5. Adolescent literacy. 

Policy and practices relevant to literacy at middle school and high school 
levels. 

6. Mathematics. 

Alignment of curriculum, instruction, and assessments with state standards in 
mathematics as well as teacher professional development in mathematics. 

7. Dropout prevention. 

Policies and practices specifically designed to keep high school students in 
school through graduation. 

8. High school 
redesign/reform. 

High school renewal, school organization, curriculum, and instructional 
practice. 

9. Transition to high school. 

Policies and practices designed to improve the preparation of middle-school 
students for success in high school. 

10. Special education- 
curriculum, instruction, 
and professional 
development. 

Curriculum, instruction, and professional development for service to students 
with disabilities. 

11. Special education- 
assessment. 

Implementation and interpretation of assessments for students with 
disabilities, including alternate assessments. 

12. English Language 
Learners. 

Curriculum, instruction, and professional development for service to ELL 
students; Title III improvement plans. This category also includes 
determination of ELLs’ achievement of AYP through assessment of Annual 
Measurable Achievement Objectives; implementation and interpretation of 
ELL assessments. 

13. Highly Qualified Teacher 
provisions of NCLB. 

State plans and policies required in connection with the HQT provisions of 
NCLB, including HOUSSE and equitable distribution. 

14. Teacher preparation and 
induction. 

State policies and practices for pre-service teacher preparation and induction 
of novice teachers as well as activities targeting teacher retention. 
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Project Inventory Form 


Topic 

Definition 

15. Teacher professional 
development. 

State policies and local practices for the planning and delivery of 
professional development for teachers (not specific to reading, mathematics, 
high schools, ELLs, or special education). Activities in the area of effective 
teaching are included in this category. 

16 . Supplemental 

Educational Services 
(SES). 

Application process to select vendors, monitoring services, and evaluating 
performance of providers. 

17. Response to Intervention 
(Rtl). 

Rtl implementation, moving Rtl into general education, Rtl as a strategy for 
school improvement. 

18. Migrant education. 

Statutorily required state Comprehensive Needs Assessment (CNA) process 
for migrant education. 

19. Indian/Native American 
education. 

Policies and practices to support the educational needs of American Indians, 
Alaska Natives, and Native Hawaiians; includes support of and compliance 
with ESEA Title VII. 

20. Data management 
compliance. 

Title I compliance, EDEN/EDFacts reporting requirements. 

21. Assessment design. 

Developing new assessments for use by clients; includes classroom, 
benchmark, and state grade-level assessments for federal accountability; 
activities may include design, item development, validation, reliability 
testing, alignment studies. 

22. Parent involvement. 

Title I and other parent involvement activities. 

23. Other 

Includes projects that do not fit under any other topic area, including 
websites, membership on SEA committees, and ongoing phone support. 
Includes projects building general skills and knowledge of leaders at all 
levels. Includes projects that assist clients with the review, collection, and 
translation of research, where there is no specific topic. If the research 
addresses a specific topic area, put the project in the appropriate row. Does 
not include internal or network meetings — these types of activities should 
not be included in the project inventory. 
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Sample Completed Project Inventory 


Center Name: 

Note: All examples in the table were taken from Year 2 baseline management plans. 


I. No. 

II. Name 

III. Description 

IV. 

Additional 

topics 

addressed? 

V. Activities and Deliverables 

VI. 

Start 

Date 

VII. 

End 

Date 

VIII. 
Major, 
moderate 
or minor 
project 

IX. 

Target 

state(s), 

region(s), 

or 

centers(s) 

X. 

Collab- 

orations 

and 

Sources: 

Other 

CCs 

XI. 

Collab- 

orations 

and 

Sources: 
Other TA 
Providers 

1. State Systems of Support for Schools Identified for Improvement 

1. 

State TA 
system for PI 
districts and 
schools 

Designed an 
integrated SEA TA 
system that reaches 
“program 
improvement” 
districts and schools 


• Convened coordinating council; 

• Created inventory of existing TA 
efforts; 

• Helped create TA plan; 

• Gave PD and TA to school support 
teams 

7/2006 

Ongoing 

Moderate 




2. 

Statewide 
system of 
support** 

Supported 
development of 
Regional School 
Service Centers as a 
system of support for 
addressing NCLB 


• Helped create a plan to guide the 
Regional Network Strategy for the 
next five to seven years; 

• Developed the next RFP for the 
RSSCs; 

• Supported RSSC implementation; 

• Developed and helped deliver PD; 

• Developed a protocol for collecting 
information on implementation 

7/2006 

6/2007 

Major 

State A 



3. 

“Significant 
Change in 
School 
Improvement 
and 

Restructuring'’ 

Developed a modular 
handbook and 
workshop on 
implementation of 
fast-paced significant 
school improvement, 
including 
restructuring 

2 (Local 
capacity) 

• Drafted 8 modules for handbook to 
be used with SEA systems of 
support; 

• Presented at institute; 

• Revised and added workshop 
materials; 

• Consulted at CC A regional 
meeting 



Major 

All 

Regional 

CCs 

Regional 

CCA 



Sample Completed Project Inventory Form 
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Support to Helped the SEA 

School School Improvement 

Improvement Team to identify 

Team resource and design 

training for schools in 
“monitored” and 
“priority” statuses 


2. Building District/Local Capacity to Support 



7/2006 6/2007 Minor 


Attended meetings organized by 
SIP team and provided reflective 
syntheses; 

Brokered resources and services 
available through RRC and CC 
networks 


School improvement, or Address Corrective Action and Restructuring 


State A 


5. 

District tools 
for monitoring 
schools 

Helped the SEA 
develop processes and 
tools for districts to 
use in monitoring the 
implementation of 
school restructuring 
plans 

6. 

Leadership 
Institute on 
Helping 
Districts Assist 
Low- 

Performing 

Schools 

Designed and 
implemented a PD 
plan to support the 
Local Education 
Agency Assistance 
Program 

7. 

Guidance to 
districts on 
restructuring 
schools 

Assisted SEA in 
developing guidance 
for districts with 
schools in 
restructuring 

3. Assessment (excluding assessment of special 

8. 

Growth 

models 

Built SEA 
understanding of 
assessment to help in 
decision making about 
accountability growth 
models 



• Prepared draft processes and tools 
for monitoring; 

• Met with SEA school improvement 
staff to review/revise tools 

7/2006 

10/2006 

Minor 

• Convened planning team to design 
the Leadership Institute; 

• Helped conduct institute; 

• Provided ongoing support to 
participants as they work with 
districts and schools 

7/2006 

6/2007 

Moderate 

• Assisted state in writing guidance 
protocol; 

• Piloted guidance; 

• Collected and prepared research; 

• Provided initial training to districts 
with schools in restructuring 

7/2006 

12/2006 

Moderate 


State A 


education students or ELL students) 


• Identified state needs and create 
plan for growth model; 

• Assisted states accepted as pilot 
states; 

• Assisted ongoing development for 
other states 



Ongoing Major 



All 

states in 
region 


Sample Completed Project Inventory Form 
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Assessment Built SEA and other 

data staff capacity by 

analysis** helping them train 

districts and schools 
in analyzing state 
assessment results and 
using them for 
instructional planning 


4. Reading/Language Arts Curriculum, Instruction, and/or Professional Development (includes Adolescent Literacy) 


• Met with SEA staff to plan PD; 

5/2006 

6/2007 

Major 

State A 

Content 

• Helped host Statewide Assessment 
Conference; 

• Continued to plan and offered 
training in assessment analysis; 

• Established a cadre of educators to 





CCA 

provide PD to districts 







Adolescent 

Literacy 

Cadres 


Organized and 
delivered training of 
adolescent literacy 
cadres made up of 
SEA staff 


• Convened SEA work teams to 
analyze research; 

• Reviewed literacy diagnostic tools; 

• Customized a literacy assessment; 

• Conducted PD for SEA work teams 
on customized assessments 


• Convened work group to discuss 
research; 

• Provided SEA with PD on Strategic 
Instruction Model; 

• Developed state literacy plan 


5. Mathematics Curriculum, Instruction, and/or Professional Development 


Developed and 
provided PD to a 
cadre of educators to 
help schools 
implement the state 
K-2 Mathematics 
Inventory 


10/200 

6 


Effective 
Leadership in 
Literacy 


K-2 

Mathematics 

Inventory 


Built SEA capacity to 
support “effective 
leadership in literacy 
for grades 6-12” 



Moderate 


Major 



All 

Content 

states in 

CC A; 

region 

Content 


CC B; 


Content 


ccc 

State A; 

Content 

State B 

CCA 




• Helped SEA design PD; 

7/2006 

6/2007 

Major 

State A 

Content 

• Helped design team pilot 
workshop; 

• Revised PD; 

• Helped deliver regional trainings 





CCA 

prioritized for schools in 
improvement; 






• Reviewed feedback to redesign PD 








Sample Completed Project Inventory Form 











































7. High School Reform 


Implementing 
High School 
Task Force 
Report 

Developed capacity of 
SEA to implement 
recommendations of 
HS Task Force Report 
related to high school 
redesign 


• Work group and Steering 
Committee met to consider task 
force recommendations, research, 
best practice, and review of prior 
initiatives; 

• Identified “knowledge 
management” components needed 
to support state implementation; 

• Identified policies and practices 
needed to promote redesign; 

• Developed multi-year 
implementation plan. 

9/2006 

4/2007 

Moderate 




8. Special Education 


14. 

Assessing 
special needs 
students 

Supported SEAs in 
establishing 
assessment and 
accountability systems 
that include and 
support special 
education, ELL, and 
low-income students 

10 (ELL) 

• Collected information from Content 
CC A on research- based practices; 

• Conducted SEA Needs 
Assessment; 

• Provided regional training(s) on 
Scientifically Based Research on 
assessment and accountability 
systems for special needs students 

7/2006 

Ongoing 

Major 

All 

states in 
region 

Content 

CCA 


15. 

Developing an 

Alternate 

Assessment 

Assisted SEAs in 
developing Alternate 
Academic 
Assessments for the 
“1” of students who 
have significant 
cognitive disabilities 

3 

(Assessmen 

t) 

• Identified needs and broker 
assistance in developing 
assessment; 

• Provided TA on development of 
assessment; 

• Provided ongoing TA on 
implementation 

7/2006 

6/2007 

Moderate 

State A; 
State B 

Content 

CCA 


16. 

Research 
Synthesis on 
Special Needs 
Students 

Published Research 
Synthesis 1 focusing 
on teaching special- 
needs students 

12 (HQT) 

• Identified and evaluated existing 
research and syntheses; 

• Conducted gap analysis to identify 
research synthesis needs; 

• Wrote synthesis (50-60 pages); 

• Posted on on-line resource forum. 

7/2006 

9/2007 

Major 

All 

Regional 

CCs 












































B-20 


9. English Language learners 

17. District Helped develop a 13 (Parents) 

guidance on comprehensive 

the education framework for 

of ELL educating English 

students Language Learners to 

guide district work, 
including guidance on 
the use of formative 
assessments to 
improve instruction 
and on family and 
community 
engagement 

18. Addressing Assisted SEA in 

AMAOs aligning Annual 

Measurable 
Achievement 
Objectives, standards, 
and assessments for 

ELL students 

10. Highly Qualified Teachers 

19. “TQ Source Developed online 

Project” resource 

clearinghouse that 
provides clients with 
policy and research 
data specifically 
related to teacher 
quality, as addressed 
in NCLB 


• Lacilitated meetings of the SEA 7/2006 6/2007 Major State A Content 

task force responsible for CC A 

developing and disseminating a 

comprehensive framework for the 
education of ELLs; 

• Assisted in integrating effective 
formative assessment practices for 
ELL students; 

• Assisted in formulating guidance 
on ELL family and community 
engagement strategies; 

• Provided consultation and 

resources from Content CC A 

• Conducted a study on alignment of 8/2006 12/2006 Minor State A; 

English Language Proficiency State B; 

standards for grades K-12 with the State C 

state assessment for ELL students 

• Helped state reconfigure AMAOs 
to align with state test 


• Developed “interactive data tool” 7/2006 Ongoing Major All 

based primarily on NCES data; Regional 

• Updated “policy database” for CCs 

state -by-state policy areas, 

including policies related to PD and 
teacher prep, recruitment, retention, 
certification, etc.; 

• Published next issue of “Tips and 
Tools: Emerging Strategies to 
Enhance Teacher Quality”; 

• Enhanced “Publications database” 
focusing on teacher quality 
research 


Sample Completed Project Inventory Form 
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20. 

Teacher 
Preparation 
and Licensure 
in literacy 

Assisted SEA in 
reviewing teacher 
preparation and 
licensure requirements 
related to literacy 

4 

(Language 

arts) 

• Assisted the Task Force on 
Licensure/Professional 
Development in revisiting 
preparation and licensure 
requirements related to literacy 
instruction for school leaders and 
teachers; 

• Discussed findings and possible 
next steps related to engagements 
with representatives from institutes 
of higher education 

10/200 

6 

2/2007 

Minor 

State A 



12. Supplemental Educational Services 

21. 

“State 

Evaluation of 

Supplemental 

Educational 

Services” 

Manual 

Updated “State 
Evaluation of SES” 
manual and assist with 
its use by Regional 
CCs 


• Drafted updates to SES evaluation 
manual 

• Presented at September 2006 
Institute 

• Revised based on evaluations and 
usage 

• Submitted evaluation report to ED 

9/2006 

4/2007 

Major 

All 

Regional 

CCs 

Regional 

CCA 


13. Other NCLB-related TA 

22. 

Enhancing 

Schoolwide 

Planning 

Materials 

Assisted SEA in 
updating resources 
and training for 
district Title I 
directors on the topics 
of schoolwide 
planning, plan 
implementation, and 
district monitoring of 
school plans 


• Helped update T1 schoolwide 
application and evaluation rubric; 

• Ensured that North Central 
Association Commission on 
Accreditation and School 
Improvement school improvement 
processes include schoolwide plan 
components; 

• Reviewed need assessment results 
from district/schools; 

• Updated materials based on needs 

7/2006 

6/2007 

Major 

State A 




**Projects nominated by the Center for review by expert panels 


Sample Completed Project Inventory Form 
































Request for Materials for Expert Panel Reviews 



Decision Information Resources, Inc. 


REQUEST FOR MATERIALS FOR EXPERT PANEL REVIEWS 


[Date] 


Dear [Center Director] : 

I am happy to report that we have completed the sampling process for your site and have 
selected the following (n) projects for review: 


Project Name 

Project Description 
















These projects will be rated for technical quality by panels of expert reviewers, and for 
relevance and usefulness through surveys of project participants. 

Assembling Panel Review Materials 

Attached you will find a Request for Materials form that includes some basic guidance 
for assembling materials for the panel review process. This form also includes the project 
cover sheet, which should be completed for each project. This cover sheet will be 
provided to the review panelists and is an important source of background and contextual 
information about the project. 

To facilitate the review process, we ask that you do the following: 

• Submit 1 hard copy and 1 electronic version of all the materials that comprise the 
review packet for each project. The hard copy version will allow us to assess whether 
the volume of material is appropriate, and will serve as a quick reference source 
should questions arise about the review packet. 
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• If an item is not available in electronic format or if you would prefer the reviewers to 
have a hard copy version of a particular item (for example, a training binder with 
multiple tabs and color handouts included), please provide 3 additional hard copies of 
that item. 

• Each review packet should include a cover sheet (using the attached template) that 
lists all of the materials included in the packet. Electronic versions of the materials 
should be filed together in a project folder that is clearly labeled with the project 
name. Feel free to use subfolders within the project folder if you think it will help 
reviewers organize the materials (for example, background/research base, meeting 
notes, training materials, etc.) in a more understandable way. You might also want to 
include the file name of each document referenced in question #5 of the cover sheet 
so that reviewers can easily sort through and match the documents on the cover sheet 
with the files they receive. 

• Please send your hard copy materials (bundled by project with a cover sheet on each), 
along with a CD containing separate folders for each of the sampled projects to: 

DIR 

Attn: 

[address] 

Materials are due no later than [date]. 

I have attached a copy of the quality scoring rubric so that you can see what the panelists 

will be looking for in the review process. As you begin to assemble the materials for 

panel review, we offer the following guidance: 

• With the exception of the cover sheet and the participant list, all materials provided in 
the review package should be materials that already exist. We do not expect you to 
create any new materials for the review process. 

• Materials included in the packet should focus on work conducted between summer 
200X and summer 200X. Materials developed before summer 200X can be included 
if they provide relevant contextual or background information. Materials developed 
after summer 200X will be considered in the next round of panel reviews (for the 
200X-200X program year), if that particular project is sampled again for review. 

• Make sure you have documented the “knowledge base” upon which the project was 
developed. As you can see in the attached rubric, this is a very important factor in the 
quality rating. Depending on the topic area and nature of the project, the knowledge 
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base might include empirical research, laws and regulations, or professional 
wisdom 81 . 

• Provide enough materials to allow reviewers to understand the project substantively, 
with a particular focus on the work conducted in the 200X-200X program year. This 
does not mean sending everything ever done or developed in connection with the 
project. You want to provide reviewers with a sufficient basis for their ratings but at 
the same time, not overwhelm them with materials. 

• Focus more on the substantive materials than on those dealing with process or 
administrative issues alone. You do not need to include every e-mail message or set 
of meeting notes on a project unless it provides critical contextual information or 
speaks directly to the technical quality of the project. For example, if you are 
assembling materials about an annual forum or institute, you should not include 
meeting notes about conference logistics or attendee evaluation results, but you 
should include notes relating to the development of the agenda and selection of 
presenters. 

If you have any questions about what materials to include in the review packet or how to 

complete the cover sheet, please contact your site visitor. 


Compiling Lists of Project Participants 

For each project selected for the evaluation, a sample of participants will receive a short, 
web-based survey from the national evaluation. This is the reason we are asking each 
Regional Comprehensive Center (RCC) and each Content Center to compile a list of all 
state-level staff who have participated in each project sampled for the evaluation. Content 
Centers should also include all RCC-based staff, as well as all state-level staff, on their 
lists of project participants. Compile a separate list of participants for each project 
sampled, allowing duplication of names across lists. 

Defining “participants.” “Project participants” include all those who have served on task 
forces and work groups associated with the project; state-level or RCC staff who have 
attended conferences, technical assistance retreats, consultations, and other meetings that 
are part of the project; and state-level or RCC staff who have received written materials 
or other resources disseminated under the project. 

If the Comprehensive Center collaborated with state-level staff to provide a service to 
others, and that collaboration was intended to build the capacity of the state in some way, 
then all of the state-level staff who collaborated with the Center on the project should be 


81 Professional wisdom is the judgment that individuals acquire through experience and is reflected through 
consensus views on decisions on implementation. Grover J. Whitehurst. Evidence-Based Education . 


2002 . 
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included on the list of project participants. These collaborators will be included in the 
pool of potential survey respondents. 

The same logic applies to Content Center projects that involve collaborations with RCC 
staff. Again, if the collaboration was intended to build the capacity of the RCC in some 
way, then all RCC staff who collaborated on the project with the Content Center should 
be included on the list of project participants and will be included in the pool of potential 
survey respondents. 

Defining “state-level.” For the purposes of this evaluation, “state-level” participants in 
Regional Center or Content Center projects may include the following: 

• State education agency (SEA) employees and consultants 

• Employees of other state agencies, such as governors’ offices 

• Employees of intermediate education agencies 82 who assist schools on behalf of 
the state 

• Individuals who serve on school support teams as part of a statewide system of 
school support under NCLB, including local educators (school district 
administrators, principals, and teachers) 

• Individuals who serve on state-level work groups or task forces, including local 
educators 

Local educators who are not serving on school support teams and are not members of a 
state-level work group or task force should not be included in project participant lists. 

RCC staff. Content Centers should include all RCC staff who have participated in each 
project, in addition to state-level project participants. Participant lists for some Content 
Center projects may include no RCC staff, some may include both RCC and state-level 
staff, and some may be made up entirely of RCC staff, depending on the nature of the 
project. 

Contact information. For each project participant, please provide a name, title (if 
available), affiliation, email address, telephone number, and regular mail address. 
Because surveys will be administered via the web, with follow-up by regular mail and by 
telephone, all of this contact information is critical. Send a separate participant list for 


S2 Intermediate education agencies are usually established by state statute, but their governance structures 
and funding sources vary from state to state. Depending on the state, they are known as Area Education 
Agencies (AEA), Boards of Cooperative Educational Services (BOCES), Cooperative Education Service 
Agencies (CESA), County Offices of Education (COE), Education Service Centers/Cooperatives (ESC), 
Education Service Districts (ESD), Regional Education Service Agency (RES A), or Regional Education 
Service Centers (RESC). Association of Educational Service Agencies, “Questions Asked About 
Educational Service Agencies,” downloaded from http://www.aesa.us/Q&ABro04.pdf on July 6, 2007. 
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each sampled project, following the template shown below (using either Word or Excel to 
create the file). 


Project name: Really Important Project 

Last 

name 

First 

name 

Title (if 
available) 

Affiliation/ 

Organization 

Email address 

Telephone 

number 

Street 

address 

City 

State 

ZIP 

Doe 

John 

Title I 
Director 

State Department of 
Public Instruction 

idoe@sea.kl2.us 

(101)555-1234 

99 State 
Street, 
Room 100 

State 

Capital 

ST 

10001 

Etc. 











Participant lists for each project should be included with your sample review materials 
and sent to DIR no later than [date]. 

For additional information. If you have any questions about the participant lists, or if 
there are special circumstances pertaining to a project or to a group of participants that 

you would like to discuss, please contact at Policy Studies Associates. 

can be reached at (xxx) xxx-xxxx or by email at @ . 

Thank you for your continued cooperation. We look forward to receiving your materials. 

Sincerely, 

Vice President 

Decision Information Resources 
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Request for Materials for Expert Panel Reviews 


National Evaluation of the Comprehensive Technical Assistance Centers 
Request for Materials for Expert Panel Review 


The projects below have been sampled from the Project Inventory Form that your Center completed for the most 
recent project year. These projects will be rated for quality by expert review panelists for the national evaluation 
of the Comprehensive Technical Assistance Centers. Surveys of participants in these projects will collect client 
ratings of the projects’ relevance and usefulness. 


Project 

Activities and Deliverables 






















In order to facilitate this process, for each project please provide us with: 

(1) Copies of the materials that together fairly represent each of these items. This should include: documents 
(electronically, wherever possible) leading up to, or resulting from, these efforts-including plans, agendas, 
meeting notes, handouts, presentations, follow-up memos, resolutions, instruments, assessments, summaries, 
syntheses, papers, reports and memoranda of agreement. In other words, we are asking for materials that fully 
describe these products and services for the reviewers and give them a sufficient basis for rating their quality. 

(2) A cover sheet (one per project) that provides background and contextual information about the project. (See 
attached page for specific questions to be answered on the cover sheet.) 

(3) A list of the state-level participants in each project, including names, affiliations, and contact information. 

Questions about what materials to provide and how to complete the cover sheet should be directed to your 

evaluation team liaison. 

These materials will be the basis upon which the reviewers will make their ratings. 


According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless it displays a valid OMB control number. The valid OMB control number for this data 
collection instrument is 1850-0823. The time required to complete these worksheets is estimated to average four 
hours per response, including the time to review instructions, search existing data sources, gather the data needed, 
and respond to the questions. Send comments regarding this burden estimate or any other aspect of this collection 
of information, including suggestions for reducing the burden, to the U.S. Department of Education, Washington, 
DC 20202-4651. 
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Request for Materials for Expert Panel Reviews 


Cover Sheet to Accompany Materials Sent to Review Panel for Sampled Projects 

Name of Comprehensive Center 

Name of person completing this form 

Telephone Email 

Information about Sampled Projects: 

Project Name 

Project Inventory # 


1. Describe the purpose of this project and its intended outcome(s). 

2. Who is the customer/client? 


3. Describe the circumstances that led to the provision of this project. In this description, 
be sure to indicate whether it was initiated by the Center or requested by the client. 

4. List the activities and deliverables associated with this project, and describe how they 
relate to the overall project. 

5. Please provide a list of the materials and documents you are sending for the review 
panel. For each, describe the Comprehensive Center’s role and contribution to the 
materials associated with this project (i.e., the materials sent to reviewers for rating 
the quality and relevance of this project.) Please let us know if you used materials 
developed by another source and, if so, provide information about that source. 

6. What is the research basis for the product or service? Where appropriate, please 
provide a citation for the research or practice literature upon which it is based. 
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Sample Follow-Up Memo 


Dear [Center Director]: 

Members of the evaluation team met in Houston last week to look over the review 
materials submitted by the Centers. In our cursory review of the PCC projects, we had 
trouble locating certain items that were either described on the project cover sheet(s) or 
that one would expect to see in the project materials. 


Project Name 

Issue Identified 








If these items were included in your review package, please let us know where in the 
materials we can locate them so that we can make sure that the peer review panelists 
will be able to find them. 

If you didn’t intend to include them in the package, please let us know. 

If you would like to send new copies of these items or additional materials to address 

the issue(s) identified, please forward copies electronically to @ for hard 

copies to [name], DIR, [address] no later than [date]. 

If you have any questions, please feel free to contact me. 

Best, 

[name] 
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Exhibit B.2. 


Distribution of all major and moderate projects and projects 
in the evaluation sample, by topic in 2006-07 


Project topic 

Number of 
major and 
moderate 
projects on 
project 
inventories 

Number of 
projects in the 
sample 
(all major or 
moderate) 

Percent of all 
major and 
moderate 
projects in 
evaluation 
sample 

Total 

216 

122 

56% 

Components of effective systems of 
support — state, district, school 

54 

36 

67 

Data use/data-driven decisionmaking 

5 

2 

40 

Formative assessment 

5 

1 

20 

Reading 

17 

7 

41 

Adolescent literacy 

11 

5 

45 

Mathematics 

11 

6 

55 

Dropout prevention 

4 

2 

50 

High school redesign/reform 

10 

5 

50 

Transition to high school 

4 

3 

75 

Special education — curriculum, instruction, 
and professional development 

5 

3 

60 

Special education — assessment 

1 

1 

100 

English language learners 

17 

11 

65 

Highly qualified teacher provisions of NCLB 

8 

5 

63 

Teacher preparation and induction 

6 

4 

67 

Teacher professional development 

8 

7 

88 

Supplemental educational services (SES) 

9 

5 

56 

Response to Intervention (Rtl) 

7 

3 

43 

Migrant education 

4 

2 

50 

Indian/native American education 

2 

1 

50 

Data management compliance 

8 

5 

63 

Assessment design 

2 

2 

100 

Parent involvement 

6 

3 

50 

Other 

12 

3 

25 


EXHIBIT READS: For program year 2006-07, there were 54 major and moderate projects on the project 
inventories that focused on components of effective systems of support. Of these, 36 (or 67 percent) were in the 
project sample. 

SOURCE: Project inventory forms submitted by the Centers. 
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Exhibit B.3. 


Distribution of all major and moderate projects and projects 
in the evaluation sample, by topic in 2007-08 


Project topic 

Number of 
major and 
moderate 
projects on 
project 
inventories 

Number of 
projects in the 
sample 
(all major or 
moderate) 

Percent of all 
major and 
moderate 
projects in 
evaluation 
sample 

Total 

222 

122 

55% 

Components of effective systems of 
support — state, district, school 

68 

39 

57 

Data use/data-driven decisionmaking 

4 

3 

75 

Formative assessment 

6 

3 

50 

Reading 

8 

2 

25 

Adolescent literacy 

15 

6 

40 

Mathematics 

8 

4 

50 

Dropout prevention 

2 

2 

100 

High school redesign/reform 

10 

6 

60 

Transition to high school 

3 

3 

100 

Special education — curriculum, instruction, 
and professional development 

3 

1 

33 

Special education — assessment 

2 

2 

100 

English language learners 

17 

11 

65 

Highly qualified teacher provisions of NCLB 

13 

7 

54 

Teacher preparation and induction 

3 

2 

67 

Teacher professional development 

6 

4 

67 

Supplemental educational services (SES) 

4 

2 

50 

Response to Intervention (Rtl) 

13 

6 

46 

Migrant education 

7 

3 

43 

Indian/native American education 

3 

0 

0 

Data management compliance 

7 

6 

86 

Assessment design 

4 

1 

25 

Parent involvement 

5 

2 

40 

Other 

11 

7 

64 


EXHIBIT READS: For program year 2007-08, there were 68 major and moderate projects on the project 
inventories that focused on components of effective systems of support. Of these, 39 (or 57 percent) were in the 
project sample. 

SOURCE: Project inventory forms submitted by the Centers. 
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Exhibit B.4. Survey of project participants sampling and survey 
administration summary, by year 



2006-07 

2007-08 

2008-09 

Sample frame (a) 

3,904 

4,011 

3,417 

Participants in RCC projects 

2,689 

1,870 

1,788 

Participants in CC projects 

1,215 

2,141 

1,629 

Study sample (b) 

1,658 

1,710 

1,364 

Sampled participants from RCC projects 




State-level participants 

1,166 

1,087 

957 

Sampled participants from CC projects 




State-level participants 

188 

296 

231 

RCC staff 

304 

327 

176 

Survey administration 




Completed surveys 

1,208 

1,319 

1,035 

Respondent participated in project 

(c) 

947 

933 

750 

Respondent did not participate in 

261 

386 

285 

project 




Refusals 

90 

11 

9 

No response 

360 

380 

320 

Response Rate 

73% 

77% 

76% 

(1,208/1,658) 

(1,310/1,710) 

(1 ,035/1 ,364) 


NOTES: Sampled respondents who indicated that they did not participate in the project were excluded from the 
analyses. To permit generalization to all participants, the survey responses were weighted for all the sampled 
projects in the following manner: total number participants for that project (a) divided by the number of respondents 
completing a survey who participated in the project (c). For project-level analyses, each respondent had equal weight 
in the individual project’s rating, and for Center-level analysis, the team weighted each sampled project from that 
Center equally. 

EXHIBIT READS: For the 2006-07 program year, the sample frame included 3,904 participants of which 2,689 were 
participants in RCC projects and 1,215 were participants in CC projects. 

SOURCE: Surveys of project participants. 
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Exhibit B.5. 


Factor analysis of relevance and usefulness items from the 
project participant surveys, by year 



2006-07 

2007-08 

2008-09 

Completed surveys 
(unweighted) 

1,208 

1,319 

1,035 

Relevance 




Eigenvalue 

5.4 

5.6 

5.7 

Percent of variance 
explained 

67.6 

69.8 

72.1 

Reliability coefficient 
(Cronbach’s Alpha) 

0.93 

0.94 

0.94 

Usefulness 




Eigenvalue 

7.6 

7.6 

7.7 

Percent of variance 
explained 

69.2 

69.3 

69.5 

Reliability coefficient 
(Cronbach’s Alpha) 

0.95 

0.95 

0.95 


EXHIBIT READS: Among the eight components of relevance, one factor was extracted with an 
eigenvalue of 5.4 and explained 67.6 percent of the variance, for the program year 2006-07. 
Among the 1 1 usefulness items, one factor was extracted for 2006-07 with an eigenvalue of 7.6 
and explained 69.2 percent of the variance. The reliability coefficient for the relevance scale was 
0.93 and it was 0.95 for usefulness. 

SOURCE: Surveys of project participants. 
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Exhibit B.6. 


Loading factors for the relevance and usefulness items from the 
project participant surveys, by year 


Relevance items 

2006-07 

2007-08 

2008-09 

Based on your experience, to what degree was this set 
of activities and resources relevant to your work, in 
each of the following respects? 




a. 

Addressed a need or problem that my organization 
faces 

0.83 

0.88 

0.87 

b. 

Addressed an important priority of my organization 

0.82 

0.84 

0.86 

c. 

Addressed a challenge that my organization faces 
related to the implementation of NCLB 

0.82 

0.78 

0.82 

d. 

Provided information, advice, and/or resources that 
could be directly applied to my organization’s work 

0.86 

0.87 

0.88 

e. 

Addressed our particular state context 

0.80 

0.83 

0.84 

f. 

Addressed my organization’s specific challenges 
(e.g., policy environment, leadership capacity, 
budget pressures, local politics) 

0.83 

0.84 

0.85 

Q- 

Provided information, advice, and/or resources that 
could be used to guide decisions about policies, 
programs, or practices 

0.85 

0.86 

0.88 

h. 

Highlighted the implications of research findings (or 
information about best practice) for policies, 
programs, or practices 

0.78 

0.77 

0.78 

Usefulness items 

2006-07 

2007-08 

2008-09 

Based on your experience, to what degree was this set 
of activities and resources useful to you, in each of the 
following respects? 




a. 

Provided resources that were easy to understand 
and easy to use 

0.82 

0.79 

0.81 

b. 

Employed an appropriate format (e.g., a work group, 
a conference, individual consultation, written 
products) 

0.81 

0.81 

0.81 

c. 

Provided adequate opportunity to learn from 
colleagues in other states 

0.64 

0.63 

0.67 

d. 

Included adequate follow-up to support the use of 
new information and resources 

0.82 

0.83 

0.84 

e. 

Were timely 

0.84 

0.83 

0.83 

f. 

Helped my organization solve a problem 

0.87 

0.88 

0.87 

9- 

Helped my organization maintain or change a policy 
or practice 

0.84 

0.84 

0.87 

h. 

Helped my organization take the next step in a 
longer-term improvement effort 

0.85 

0.88 

0.85 

i. 

Provided my organization with information or 
resources that we will use again 

0.89 

0.89 

0.89 

j- 

Helped my organization develop a shared expertise 
or knowledge base 

0.88 

0.88 

0.87 

k. 

Helped individuals in my organization to develop 
skills that they will use again 

0.87 

0.88 

0.85 
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Appendix C 

Describing Center Operations 




Describing Center Operations 


This appendix contains a description of the process used to code the objectives within 
Center management plans, the project inventory forms (PIFs), and the project cover sheets. This 
appendix also includes the protocols used for the interviews with the Centers in person in the 
summers of 2007 and 2008 and by phone in Spring 2010. 


Center Management Plan, Project Inventory, and Cover Sheet 
Coding and Intercoder Reliability 

Data from the Center management plans, project inventory forms (PIFs), and the project 
cover sheets were coded by the evaluation team to build the sample frame and describe Center 
operations. Below is an explanation of the four-step coding process the evaluation team used-(l) 
establish the code sets, (2) train coders, (3) code management plans, PIFs, and cover sheets, (4) 
resolve coding disagreements-as well as a summary of the level of intercoder reliability achieved 
through this process. 


Establish the Code Sets 

Based on initial review of the project cover sheets, the PIFs, and the other materials from 
the Centers (such as the Center management plans), the evaluation team established code sets to 
use during the coding process. One set of codes was used to code the objectives stated in the 
Center management plans and the topics of the projects on the PIFs (for 2006-07). Another set of 
codes was used to code the project cover sheets and a third set of codes was used in a separate 
effort to recode the 33 project cover sheets for the 2008-09 state systems of support projects. 


Center Management Plan and PIF Code Set 

The team developed codes to summarize the primary topics in which Centers were doing 
their work based on a review of Center materials to identify categories of distinct domains or 
topics. These 22 specific topic codes are described below in terms of the substantive knowledge 

O'} 

base the evaluation team determined would be covered by that topic. In the coding process, this 
set of possible codes was used to code the topic areas for the Center objectives as stated in their 
management plans as well as to code the topic area of each project that Centers submitted on 
their PIFs for 2006-07. 

■ Components of effective systems of support-state, district, school. This category 
included: (1) Design of state systems to meet NCLB requirements for statewide 
systems of support for districts and schools in corrective action or identified as in 


83 Discussion in the report refers primarily to the 22 specific substantive topics. However, as a small number of 
sampled projects fell into the category of “other” topic area, description of all the projects in the study sample by 
topic as displayed in exhibit 2.5 includes the topic area of “other” as the 23 rd category. 
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need of improvement [Sec. 1117(a)]; (2) Interventions for schools or districts in 
need of improvement or corrective action; (3) For schools, Centers' projects 
addressed a range of options for school leadership and organization, professional 
development and classroom practice, and development and implementation of 
school improvement plans. For districts, Centers' projects typically addressed 
district leadership and development as well as implementation of district 
improvement plans. 

■ Data use/data-driven decisionmaking. Use of data for educational program 
improvement. 

■ Formative assessment. Guidelines for implementing formative or benchmark 
assessment, comprehensive formative assessment systems. 

■ Reading. Alignment of curriculum, instruction, and assessments with state 
standards in reading; teacher professional development in reading (not specific to 
adolescent literacy). 

■ Adolescent literacy. Policy and practices relevant to literacy at middle school and 
high school levels. 

■ Mathematics. Alignment of curriculum, instruction, and assessments with state 
standards in mathematics as well as teacher professional development in 
mathematics. 

■ Dropout prevention. Policies and practices specifically designed to keep high 
school students in school through graduation. 

■ High school redesign/reform. High school renewal, school organization, 
curriculum, and instructional practice. 

■ Transition to high school. Policies and practices designed to improve the 
preparation of middle- school students for success in high school. 

■ Special education-curriculum, instruction, and professional development. 
Curriculum, instruction, and professional development for service to students with 
disabilities. 

■ Special education-assessment. Implementation and interpretation of assessments 
for students with disabilities, including alternate assessments. 

■ English language learners. Curriculum, instruction, and professional 
development for service to ELL students; Title III improvement plans. This 
category also includes determination of ELLs’ achievement of AYP through 
assessment of Annual Measurable Achievement Objectives; implementation and 
interpretation of ELL assessments. 

■ Highly qualified teacher provisions of NC LB. State plans and policies required 
in connection with the HQT provisions of NCLB, including HOUSSE and 
equitable distribution. 

■ Teacher preparation and induction. State policies and practices for pre-service 
teacher preparation and induction of novice teachers as well as activities targeting 
teacher retention. 

■ Teacher professional development. State policies and local practices for the 
planning and delivery of professional development for teachers (not specific to 
reading, mathematics, high schools, ELLs, or special education). Activities in the 
area of effective teaching are included in this category. 
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■ Supplemental educational Services (SES). Application process to select vendors, 
monitoring services, and evaluating performance of providers. 

■ Response to Intervention (Rtl). Rtl implementation, moving Rtl into general 
education, Rtl as a strategy for school improvement. 

■ Migrant education. Statutorily required state Comprehensive Needs Assessment 
(CNA) process for migrant education. 

■ Indian/native American education. Policies and practices to support the 
educational needs of American Indians, Alaska Natives, and Native Hawaiians; 
includes support of and compliance with ESEA Title VII. 

■ Data management compliance. Title I compliance, EDEN/EDFacts reporting 
requirements. 

■ Assessment design. Developing new assessments for use by clients; includes 
classroom, benchmark, and state grade-level assessments for federal 
accountability; activities may include design, item development, validation, 
reliability testing, alignment studies. 

■ Parent involvement. Title I and other parent involvement activities. 


Cover Sheet Code Sets 

The team used the information contained in the project cover sheets in two ways. First, in 
each of the three years, the team developed codes to describe project activities and resources, 
contributions CCs made to RCC projects, and contributions RCCs made to CC projects. The 
development of these codes was informed by the team’s review of the Center management plans, 
the transcripts of the interviews conducted in 2007, and a preliminary review of all project cover 
sheets. The code sets used to code the project cover sheets for all three data collection cycles are 
shown below along with the criteria used to assign that code. The 15 possible codes for this 
effort are provided below in exhibit C.l. 

Second, for 2008-09, the team conducted a supplemental analysis of the project cover 
sheets for the 33 sampled projects under the topic State Systems of Support. For these projects, 
the team went into greater depth regarding the components of the project to more closely 
examine their content and purpose. The five possible codes for this effort are provided in exhibit 
C.2. 
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Exhibit C.l. Project cover sheet codes used across the sampled projects in all 

three data collection cycles 


Types of Project Activities and Resources 

■ Engagement of participants in project planning was coded for projects that offered some or all 
of their participants the opportunity to take part in designing the project. This involvement went 
beyond merely expressing a need for technical assistance; rather, it included participant 
involvement in shaping the specific assistance that was delivered (who did what, when, and why 
in technical assistance) at various stages of technical assistance delivery). 

Project planning was different from “ongoing consultation and follow-up,” though the same project 
may have had both components. “Ongoing consultation and follow-up” occurred when the Center 
provided service to clients on multiple occasions, whereas “engagement of participants in project 
planning” was specific to the Center working with the client to plan later service. 

■ Research collections or syntheses were research-based materials distributed to participants in 
a project. In some instances these products were developed by the Center conducting the project, 
but in others they were existing research materials that the Center collected and distributed. 

■ Training events were designed to impart skills to participants, equipping them to carry out a 
particular program or strategy. The content of trainings was discrete and specialized, and 
participants attended trainings expecting to learn something specific and be able to do something 
differently afterward. Participants and trainers assumed that the primary purpose of the training 
was for the participant to learn something from the trainer. Projects with multiple trainings to the 
same participants over time were also coded as “ongoing consultation and follow-up.” 

■ Conferences featured presentations of information to audiences from multiple speakers or 
discussants representing a range of perspectives. They focused on informing the participants, not 
training them, and they were single events rather than series of meetings. The conference scope 
was broader than trainings, in that presentations or discussions in a single conference could have 
addressed multiple issues, programs, or solutions; whereas a training focused on a single 
strategy or program. Conference participants varied in their type, interest, and expectation for 
what they wanted to learn at the conference. Conferences also included symposia and forums in 
which participants came at common questions with varying perspectives. 

■ Support for development of a formal plan to implement a program or policy referred to 
instances in which Center technical assistance supported work by participants that led to a plan 
for implementing a program or policy. A formal state plan codified activities that were expected to 
occur in the state in order to serve state purposes with the intention to guide individuals and units 
toward a common goal. Formal plans may have been required by the federal or state 
government, or may have been initiated within the SEA to address a specific problem. This 
activity did not include service plans. 

■ Task force meetings and work were engagements over multiple interactions in which a Center 
provided technical assistance to a work group who were themselves constituted as a task force to 
address a state problem. Center support of a task force meant that the Center was a regular 
participant in task force meetings, and Center support ranged from setting the agenda and 
facilitating meetings to providing input upon request. A project was coded as “task force meetings 
and work,” when there was a task force that was trying to accomplish some purpose — not just to 
plan or follow up on Center technical assistance, but to do something for its state (or, 
occasionally, a group of states). Center participation in a task force was also coded “ongoing 
consultation and follow-up” if the Center provided assistance on an ongoing basis over time. If the 
Center was working with the task force to plan service, the project was also coded “engagement 
of participants in project planning.” 
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Exhibit C.l. Project cover sheet codes used across the sampled projects in all 

three data collection cycles (continued) 


■ Ongoing consultation and follow-up was either a series of consultations (i.e., fulfilling repeated 
requests from participants for technical assistance) or sustained follow-up with individuals who 
had participated in some other kind of assistance (e.g., a conference or training). Ongoing 
consultation occurred when there were multiple service contacts over time, all related to the 
project goal or topic. The interactions focused on a client question or need. This code did not 
include projects with one-time contacts, such as projects that provided a large conference with no 
follow-up. It also did not include the production of publications that are not accompanied with 
follow-up assistance directly to the client. 


RCC and CC Contributions to RCC and CC Projects 

m RCC projects with CC contribution were projects in which the CC explicitly contributed a 

service or resource used by the RCC or its clients as part of RCC project activities. A CC could 
contribute to an RCC project in any of three ways: 

■ CC provided materials used in this RCC project. The RCC must have used 

CC materials in specific delivery activities or in planning. 


CC provided advice to the RCC on this project. The RCC consulted or 
collaborated with the CC when making decisions about what or how to deliver assistance 
to project participants. Participants may or may not have had direct interaction with the 
CC or CC materials. 


■ CC delivered technical assistance to RCC project participants. The CC 

provided service directly to the participants, whether face-to-face, by phone, or virtually, 
regardless of what role RCC staff had in this RCC project activity. 

■ CC projects with RCC contribution were projects in which the RCC explicitly contributed 

assistance used by the CC or its clients as part of CC project activities. An RCC could contribute 
to a CC project in either of two ways: 

■ RCC recruited participants or brokered service. The RCC connected the CC 
and its clients for the purpose of participation in the project. The RCC may have been 
driven by the CC’s need for participants who could benefit from a specific project, or by a 
client’s need for specific services or products that a CC could deliver. 


RCC delivered technical assistance to CC project participants. The RCC 

provided service directly to participants, whether face to face, by phone, or virtually, 
regardless of what role CC staff had in this CC project activity. 
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Exhibit C.2. Project cover sheet codes for the 33 state systems of support 

projects in 2008-09 


Purposes of State Systems of Support Projects 

m Redesign state SSOS infrastructure or strategies for improvement included projects in which 
the Center worked with SEA leaders who were making decisions about revising design features 
of the state’s SSOS. These projects may have assisted the SEA in making changes to structural 
aspects of the SSOS, such as reorganizing units within the SEA, redefining the responsibilities of 
specific SSOS support agents, and clarifying the processes and timelines associated with 
delivery. These projects also may have assisted the SEA in making changes to the strategies the 
SSOS and its agents deliver to districts and schools to guide their improvement, such as 
intervention models, promising practices based on research, steps for improvement planning, 
indicators of improvement, and other content delivered by school support teams. 

Review of the state’s current SSOS included projects in which the Center helped the SEA review or 

assess the existing system as a step in a plan for its improvement. These projects could include a 
variety of approaches for reviewing SSOS systems, including strategic reviews with SEA task 
forces, needs sensing interviews with SEA leaders and support agents, surveys of districts and 
schools, development of indicators of system functioning, and a formal CC-developed process for 
evaluating the SSOS. 

Deliver a formal SSOS needs assessment report to the SEA was coded when there was evidence that 
the Center, after conducting a needs assessment of evaluation of the SSOS, provided SEA 
leaders with a formal, state-specific, report including what was learned and implications for 
system redesign. 

Develop state-specific SSOS materials included projects in which the Center assisted SEAs in 
developing their own materials for use in SSOS implementation. These materials included 
protocols for SSOS agents to use in the field, tools for monitoring programs, SSOS training and 
communication resources, tools for improvement planning, and state plans and policy documents 

■ Train SSOS support providers included projects in which the Center assisted the state in 

training, informing, or coaching school support teams or other SSOS agents on how to carry out 
their functions in supporting low-performing districts and schools. These projects may have 
provided training on improvement strategies and models endorsed by the state, as well as on 
how providers might best deliver these strategies and models. For instance, trainings may review 
templates and guidance documents for local improvement planning and implementation. 

Trainings may have also delineated the responsibilities and procedures of support providers and 
how various SSOS components fit together as a system. 


Train Coders 

For each data collection and reporting cycle, all coders were senior members of the 
evaluation team and were trained to use the established code sets. As part of the training, coders 
discussed each of the codes in the code set to ensure all coders had a shared understanding of 
their meaning. They also coded sample items and reviewed, coded, and discussed the final codes 
using representative sample text. The evaluation team used the discussions in the coder training 
to verify that the established codes were clear and well understood. 


C-6 





Code Center Management Plans, PIFs, and Cover Sheets 


All of the objectives in the Center management plans, the PIFs, and cover sheets were 
independently coded by two coders. For all three years, to code the Center management plans, 
each coder reviewed the objectives section of the plan and determined the topic(s) in which the 
Center indicated they planned to work. For the PIFs in 2006-07, each coder reviewed the project 
description and assigned the project subject code for each of the projects (364 projects total). For 
all three years, in order to code the cover sheets, each coder categorized the project information 
submitted by Centers (i.e., project purpose, customer/client, project activities and deliverables, 
roles and contributions of Centers, research basis for products and services) and to assign codes 
according to the coding criteria described earlier (see exhibit C.l starting on page C-4) for 
identifying the various types of project activities and resources and the ways Centers contributed 
to projects. For the 2008-09 program year, the evaluation team executed a second coding effort 
of the project cover sheets for the 33 projects classified in the topic area state systems of support 
in the general areas of Center activities on SSOS projects to more closely define the purpose of 
this subset of projects (see exhibit C.2 on page C-6). 


Resolve Coding Disagreements 

Members of the evaluation team compared the coding worksheets from each coder to 
determine the final coded responses for each item. In cases where the coders agreed, the final 
response was clear. In cases where the first and second coders did not arrive at the same code, 
they met to review evidence for the codes and choose a final code. If the first and second coder 
were unable to reconcile their codes, a third senior study team member worked with the coders to 
determine the final code for that item. Each year, the product of this step was the final codes for 
the topic areas included in the Center management plan objectives, the topic areas for the all 
projects on the PIFs, and the three areas described above for the project cover sheets. 


Analysis of Intercoder Reliability 

The evaluation team calculated the percent agreement between the first and second coder 
as well as Cohen’s Kappa analysis. The percent agreement calculation used the number of 
agreements divided by total number of possible responses. The Cohen’s Kappa calculation used 
the number observed to agree minus the number expected to agree by chance divided by the 
number of items minus the number expected to agree. The exhibit below shows the results of 
these analyses. 
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Exhibit C.3. Summary of intercoder reliability for coding, by year 


Coding effort 

2006-07 

2007-08 

2008-09 

Center management plans 




Possible topic area assignments 

462 

462 

462 

Number of agreements 

452 

433 

449 

Cohen’s Kappa 

.96 

.91 

.93 

Project inventory forms (PIFs) 




Possible topic area assignments 

364 

NA 

NA 

Number of agreements 

318 



Cohen’s Kappa 

.92 




Project cover sheets 

Possible topic area assignments 1,830 1,830 1,770 

Number of agreements 1,590 1,473 1,451 

Cohen’s Kappa .89 .79 .81 


Supplemental SSOS coding for 2008-09 


Possible topic area assignments 

NA 

NA 

165 

Number of agreements 



136 

Cohen’s Kappa 



.84 


NOTE: The evaluation team only coded the topics for the project inventory forms for the 
2006-07 projects. For the subsequent data collection cycles, the Centers submitted their 
PIFs with the projects already coded by topic. The additional coding effort for the SSOS 
projects was done only for the 2008-09 projects. 

EXHIBIT READS: Across all three data collection cycles, there were a total of 462 
possible topic area assignments. In 2006-07, the first and second coder agreed in 452 of 
those cases with a Cohen's Kappa of .96. 


Center Interviews 

The evaluation team conducted three rounds of interviews with the Center Director and 
other senior Center staff to gather information about the Center operations. The first round of 
interviews covered the 2006-07 program year and took place in summer 2007. These interviews 
were based on open-ended questions posed to RCC and CC staff about Center organization, 
major areas of focus, communication with client organizations, sources of knowledge used, 
approaches taken in quality assurance, modes of delivering technical assistance, and barriers they 
encountered. The second round of interviews took place in summer 2008 and used a protocol that 
included closed-ended, binary questions to follow up specifically about activities that had 
occurred in 2006-07, topics that had been addressed, and work requests that had fallen outside 
their planned scope of work. Additional questions targeted the strategies that Centers had used 
for planning work and engaging clients, the sources for content expertise, and sources they used 
for vetting their products and services. For the third round of interviews, the team asked the 
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Centers to provide information on Center operations for 2008-09 on their strategies and 
processes for planning and negotiating work with clients, the types of activities most frequently 
requested by their clients, quality assurance, coordination with other Centers, and the evolution 
of their work from 2006 through 2009. The protocol used to interview CC staff for the third 
round of data collection starts on page C-10 of this appendix, the protocol used to interview RCC 
staff starts on page C-15. 
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2008-09 CC Interview Protocol 


Content Center Interview Protocol, 2008-09 


For these questions, please refer to the Project Inventory Form for 2008-09 as necessary to 
refresh your memory on the full scope of work delivered by your Center in that year. 


Planning and Negotiating Work 

1. Which of the following were part of the process of planning your work for the 2008-09 year: 

a. Assessment of state needs 

i. Through formal needs assessment, such as a survey 

ii. Through ongoing interaction with states 

iii. Through communication with RCCs about state needs 

b. Assessment of RCC needs 

i. Through formal needs assessment, such as a survey 

ii. Through ongoing interaction with RCCs 

c. Communication with RCC directors 

d. Receiving requests from ED 

e. Other source(s) of input, please describe: 

2. What were your top strategies for building or maintaining working relationships with RCCs 
and/or with state(s) for the 2008-09 year, so that your Center could meet its technical 
assistance objectives? Please describe specifically how you carried out these strategies. 

3. Were there instances of your Center receiving a technical-assistance request for 2008-09 
from an RCC or a state that the Center did not carry out? Please describe the types of 
request(s). If there were multiple instances, describe one that was most typical of your 
experience. 

4. For 2008-09, did the reasons for not carrying out technical-assistance requests ever include 
the following: 

a. A request fell outside the legitimate scope of work for a Content Center 

b. A request was potentially legitimate Content Center work but did not fit this Center’s 
priorities for work with the client(s) 

c. The Center did not have access to the needed expertise to carry out the request 

d. Staff time and resources were already fully committed to other work 

e. The request became moot because the Center and the client instead agreed on a plan for 
related, but different, technical assistance 

f. Other reason(s), please describe 

5. Of the reasons you’ve just identified, which was the one most frequent reason requests were 
not carried out? 

6. Were there instances of your Center proposing technical-assistance activities for 2008-09 that 
RCCs, states, or ED declined to accept? Please describe the proposed technical assistance 
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2008-09 CC Interview Protocol 


and why it was not carried out. If there were multiple instances, describe one that was most 
typical of your experience. 

7. Were any of the following a barrier to delivering technical assistance that met your Center’s 
objectives for work with RCCs during 2008-09? 

a. The process of negotiating a work scope and organizing projects took too long 

b. RCCs’ most important priorities for assistance fell outside the Center’s scope of work 

c. RCCs secured most of the technical assistance they needed from other sources 

d. RCCs would have preferred to locate and contract directly with experts or consultants 
rather than working with the Content Center 

e. A policy or priority shift at the state level caused the Center’s assistance to an RCC to be 
less helpful than it might 

f. Center staff were not able to spend as much time working with an RCC as the RCC 
would have liked 

g. The Center did not have the expertise an RCC needed 

h. The Center was unable to develop a productive working relationship with an RCC 

i. An RCC experienced turnover in leadership 

j . An RCC experienced turnover in staffing 

k. RCCs placed a higher priority on completing short-term work than on addressing long- 
term purposes 

l. RCC staff did not have time to work with the Center 

m. There was a lack of coordination or communication within an RCC 

n. Other(s), specify: 

8. Now, using the same list including any other items that you added, please identify the one 
barrier in working with RCCs that most impeded the achievement of your Center’s 
objectives, and describe an example. 
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2008-09 CC Interview Protocol 


9. Were any of the following a barrier to delivering technical assistance that met your Center’s 
objectives for work with states during 2008-09? 

a. The process of negotiating a work scope and organizing projects took too long 

b. States’ most important priorities for assistance fell outside the Center’s scope of work 

c. States secured most of the technical assistance they needed from other sources 

d. States would have preferred to locate and contract directly with experts or consultants 
rather than working with the Content Center 

e. A policy or priority shift at the state level caused the Center’s assistance to a state to be 
less helpful than it might 

f. Center staff were not able to spend as much time working with a state as the state would 
have liked 

g. The Center did not have the expertise a state needed 

h. The Center was unable to develop a productive working relationship with a state 

i. A state agency experienced turnover in leadership 

j. A state office, division, or intermediary unit experienced turnover in staffing 

k. States placed a higher priority on completing short-term work than on addressing long- 
term purposes 

l. State staff did not have time to work with the Center 

m. There was a lack of coordination or communication within a state 

n. Other(s), specify: 

10. Now, using the same list including any other items that you added, please identify the one 
barrier in working with states that most impeded the achievement of your Center’s 
objectives, and describe an example. 
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2008-09 CC Interview Protocol 


The Center’s Work During 2008-09 

11. First, I’d like to know about the types of activities and products that your clients asked for 
and that your Center provided during the program year from July 2008 through June 2009. 
Looking at the following types of activities and products, please identify the one type that 
was most requested by RCCs and states during that year. Which one type represented the 
largest investment of Center resources? Which one was most important for achieving your 
Center’s objectives? Were there changes between 2006-07 and 2008-09 in which type was 
most requested, or your largest investment, or the most important type? [If yes,] Please 
describe. 

a. Engagement of participants in project planning (more than needs assessment or 
identifying participants) 

b. Research collections and syntheses 

c. Conferences (also called symposiums, forums, institutes; highlight a range of 
perspectives, strategies, or programs) 

d. Training events (focused on implementing a specific program or strategy) 

e. Task force meetings and work (focused on addressing a specific problem, program or 
policy) 

f. Support for development of a formal state plan to implement a program or policy 

g. Ongoing consultation and follow-up (multiple contacts with the same participants, that 
were part of a coherent and purposeful whole) 

12. Now I’d like to know about continuity and change in your technical assistance activities over 
the years. Considering all of your Center’s work in 2008-09, which one of the following 
descriptions would characterize the largest share of the total investment of Center resources? 
Please describe an example of this type of work. Was there a change between 2006-07 and 
2008-09 in which type was the largest investment? [If yes,] Please describe. 

a. Multi-year project(s) that followed a long-term plan for participants, activities, and 
purposes 

b. Multi-year project(s) with substantial shifts from the originally planned participants, 
activities, or purposes 

c. Projects that were self-contained within a single year rather than extending across years 


Quality Assurance 

13. For quality assurance in technical assistance during 2008-09, did your Center have a formal 
process? [If yes,] Please describe that process. 
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Coordination with Regional Centers 

14. In which of the following ways did your Center work with at least some of the Regional 
Centers during 2008-09: 

a. Maintained communication with liaisons at least monthly 

b. Provided knowledge resources (materials or experts) as requested 

c. Teamed up to work with one or more states in a region 

d. Sought information from Regional Centers about promising practices in their regions 

15.1 would like know what strengths and weaknesses you observed in the two-tier system of 
technical assistance from Regional Centers and Content Centers as of the 2008-09 program 
year. Below are possible areas of strength or weakness, and others may come to mind. Please 
identify any of these that were areas of strength in the two-tier Comprehensive Center 
network. Were any of them areas of weakness of the network? Please give examples that 
illustrate strengths and weaknesses. 

a. Clarity in the role of Regional Centers 

b. Clarity in the role of Content Centers 

c. Coordination across Regional and Content Centers 

d. Other area(s) of strength or weakness, please describe: 


Evolution of the Center’s Work 

16. Looking back, what would you describe as the most significant ways in which your Center’s 
work evolved from 2006 through 2009? 


C-14 



2008-09 RCC Interview Protocol 


RCC Interview Protocol, 2008-09 


For these questions, please refer to the Project Inventory Form for 2008-09 as necessary to 
refresh your memory on the full scope of work delivered by your Center in that year. 


Planning and Negotiating Work with State Agencies 

17. Did your Center have one or more staff positions designated for serving as the Center’s 
liaison to specific SEAs for the 2008-09 year? If so, did any of them maintain office space 
within an SEA building? 

18. Which of the following were part of the process of planning your work for the 2008-09 year: 

a. Formal needs assessment, such as a survey of client staff 

b. Informal needs assessment through communication with client staff 

c. Communication with chief state school officer(s) 

d. Other source(s) of input, please describe: 

19. What were your top strategies for building or maintaining working relationships with state(s) 
in your region for the 2008-09 year, so that your Center could achieve its technical assistance 
objectives? Please describe specifically how you carried out these strategies. 

20. Were there instances of an SEA making a technical assistance request for 2008-09 that the 
Center did not carry out? Please describe the requests that were not carried out, state by state. 
What types of requests were most common, if any? 

21. For 2008-09, did the Center’s reasons for not carrying out SEA technical assistance requests 
ever include the following: 

a. A request fell outside the legitimate scope of work for a Comprehensive Center 

b. A request was potentially legitimate Comprehensive Center work but did not fit this 
Center’s priorities for work with the state 

c. The Center did not have access to the needed expertise to carry out the request 

d. Staff time and resources were already fully committed to other work 

e. An SEA request became moot because the Center and SEA instead agreed on a plan for 
related, but different, technical assistance 

f. Other reason(s), please describe: 

22. Of the reasons you’ve just identified, which was the one most frequent reason SEA requests 
were not carried out? 

23. Were there instances of the Center proposing technical assistance activities for 2008-09 that 
an SEA declined to accept? Please describe the proposed technical assistance and why it was 
not carried out, state by state. 

24. Were any of the following a barrier to delivering technical assistance that met your Center’s 
objectives during 2008-09? 
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a. The process of negotiating a work scope and organizing projects took too long 

b. A state’s most important priorities for assistance fell outside the Center’s scope of work 

c. A state secured most of the technical assistance it needed from other sources 

d. A state would have preferred to locate and contract directly with experts or consultants 
rather than working with the Comprehensive Centers 

e. A policy or priority shift at the state level caused the Center’s assistance to be less helpful 
than it might 

f. Center staff were not able to spend as much time working with a state as the state would 
have liked 

g. The Center did not have the expertise a state needed 

h. The Center was unable to develop a productive working relationship with a state 

i. A state experienced turnover in leadership 

j. A state office, division, or intermediary unit experienced turnover in staffing 

k. State clients placed a higher priority on completing short-term work than on addressing 
long-term purposes 

l. State staff did not have time to work with the Center 

m. There was a lack of coordination or communication within a state agency 

n. Other barrier(s), please describe: 

25. Now, using the same list including any other items that you added, please identify the one 
barrier that most impeded the achievement of your Center’s objectives, and describe an 
example of this barrier. 


The Center’s Work During 2008-09 

26. First, I’d like to know about the types of activities and products that your clients asked for 
and that your Center provided during the program year from July 2008 through June 2009. 
Looking at the following types of activities and products, please identify the one type that 
was most requested by clients during that year. Which one type represented the largest 
investment of Center resources? Which one was most important for achieving your Center’s 
objectives? Were there changes between 2006-07 and 2008-09 in which type was most 
requested, or your largest investment, or the most important type? [If yes,] Please describe. 

a. Engagement of participants in project planning (more than needs assessment or 
identifying participants) 

b. Research collections and syntheses 

c. Conferences (also called symposiums, forums, institutes; highlight a range of 
perspectives, strategies, or programs) 

d. Training events (focused on implementing a specific program or strategy) 

e. Task force meetings and work (focused on addressing a specific problem, program or 
policy) 

f. Support for development of a formal state plan to implement a program or policy 

g. Ongoing consultation and follow-up (multiple contacts with the same participants, that 
were part of a coherent and purposeful whole) 

27. Now I’d like to know about continuity and change in your technical assistance over the years. 
Considering all of your Center’s work in 2008-09, which one of the following descriptions 
would characterize the largest investment of Center resources? Please describe an example of 
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this type of work. Was there a change between 2006-07 and 2008-09 in which type was the 
largest investment? [If yes,] Please describe. 

a. Multi-year project(s) that followed a long-term plan for participants, activities, and 
purposes 

b. Multi-year project(s) with substantial shifts from the originally planned participants, 
activities, or purposes 

c. Projects that were self-contained within a single year rather than extending across years 


Quality Assurance 

28. For quality assurance in technical assistance during 2008-09, did your Center have a formal 
process? [If yes,] Please describe that process. 


Coordination with Content Centers 

29. With which of the Content Centers did your Center do each of the following during 2008-09: 

a. Maintained communication at least monthly through a liaison 

i. AACC 

ii. CII 

iii. COI 

iv. NCCTQ 

v. NHSC 

b. Received knowledge resources (materials or experts) that your Center requested 

i. AACC 

ii. CII 

iii. COI 

iv. NCCTQ 

v. NHSC 

c. Teamed up to work with one or more states in the region 

i. AACC 

ii. CII 

iii. COI 

iv. NCCTQ 

v. NHSC 

d. Informed the Content Center about promising practices in the region 

i. AACC 

ii. CII 

iii. COI 

iv. NCCTQ 

v. NHSC 
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30. Were any of the following a barrier to your Center receiving technical assistance from the 
Content Centers during 2008-09? 

a. The process of negotiating a work scope and organizing projects took too long 

b. Our most important priorities for assistance fell outside the Content Centers’ scope of 
work 

c. We secured most of the technical assistance we needed from other sources 

d. We would have preferred to locate and contract directly with experts or consultants rather 
than working with Content Centers 

e. A policy or priority shift at the state level caused Content Centers’ assistance to us to be 
less helpful than it might 

f. Content Center staff were not able to spend as much time working with us as we would 
have liked 

g. Content Centers did not have the expertise we needed 

h. Our Center was unable to develop a productive working relationship with a Content 
Center 

i. We experienced turnover in leadership 

j. We experienced turnover in staffing 

k. We placed a higher priority on completing short-term work than on addressing long-term 
purposes 

l. Our staff did not have time to work with a Content Center 

m. There was a lack of coordination or communication within our Center 

n. Other(s), specify: 

31. Now, using the same list including any other items that you added, please identify the one 
barrier that most impeded your Center receiving technical assistance from Content Centers. 

32. 1 would like know what strengths and weaknesses you observed in the two-tier system of 
technical assistance from Regional Centers and Content Centers as of the 2008-09 program 
year. Below are possible areas of strength or weakness, and others may come to mind. Please 
identify any of these that were areas of strength in the two-tier Comprehensive Center 
network. Were any of them areas of weakness of the network? Please give examples that 
illustrate strengths and weaknesses. 

a. Clarity in the role of Regional Centers 

b. Clarity in the role of Content Centers 

c. Coordination across Regional and Content Centers 

d. Other area(s) of strength or weakness, please describe: 


Evolution of the Center’s Work 

33. Looking back, what would you describe as the most significant ways in which your Center’s 
work evolved from 2006 through 2009? 
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Center Operations - Supplemental Tables 


This appendix serves as a supplement to the information presented in chapter 3 of the 
report and includes tables from the 2006-07 program year that parallel similar 2008-09 tables 
included in the report. 




Exhibit D.l. Topics on which Centers stated objectives and/or delivered 
projects, 2006-07 




Number of Centers 

(N=21) 


Topic 

Objective on 
topic was 
stated in 
management 
plan 

(D 

Objective on topic was 
stated in management plan 
AND at least one 
project on topic was 
reported on PIF 
(2) 

At least one 
project on 
topic was 
reported on 
PIF 
(3) 

Total cases of a Center setting an 
objective and/or reporting a project 

133 

107 

172 

Components of effective systems of 
support — state, district, school 

15 

15 

17 

English language learners 

10 

10 

15 

Highly qualified teacher provisions of NCLB 

8 

5 

10 

Teacher professional development 

6 

5 

8 

Teacher preparation and induction 

5 

3 

5 

Data use/data-driven decisionmaking 

8 

4 

7 

Assessment design 

6 

5 

5 

Formative assessment 

5 

4 

7 

Special education — curriculum, instruction, 
and professional development 

8 

3 

6 

Response to Intervention (Rtl) 

3 

3 

10 

Special education — assessment 

3 

3 

3 

High school redesign/reform 

8 

7 

10 

Transition to high school 

2 

2 

5 

Dropout prevention 

2 

2 

2 

Mathematics 

7 

4 

8 

Adolescent literacy 

6 

6 

8 

Reading 

6 

5 

8 

Supplemental educational services (SES) 

7 

7 

12 

Parent involvement 

7 

7 

11 

Migrant education 

5 

3 

6 

Data management compliance 

5 

3 

5 

Indian/Native American education 

1 

1 

4 


NOTE: Column (1) - column (2) =the number of Centers that deleted work in the topic area as indicated by number of Centers 
that had a topic-related objective in their management plan minus the number of Centers that had the topic area on their PIF 
and their management plan; column (3)- column (2) =the number of Centers that added work in the topic area as indicated by 
number of Centers that reported work on their PIF minus the number of Centers that had the topic area on their PIF and their 
management plan. 

EXHIBIT READS: For the topic “Components of effective systems of support — state, district, school,” 15 Centers had a related 
objective in their respective annual management plans; of these, all 15 reported projects on the topic in their PIF; and a total of 
17 Centers had projects on the topic, whether or not they had stated an objective related to it in their management plan. 
SOURCES: Center management plans for 2006-07; PIFs prepared by Centers in consultation with evaluation team. 
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Exhibit D.2. Topics on which RCCs and CCs stated objectives and/or delivered 
projects, 2006-07 




Number of RCCs (N= 

16) 


Number of CCs (N=5) 



Objective 
on topic in 
manage- 
ment plan 
(1) 

Objective on 
topic in plan AND 
project(s) on 
topic on PIF 
(2) 

Project(s) 
on topic on 
PIF 

(3) 

Objective 
on topic in 
manage- 
ment plan 

(4) 

Objective on 
topic in plan AND 
project(s) on 
topic on PIF 
(5) 

Project(s) 
on topic on 
PIF 

(6) 

Total cases 

112 

89 

146 

21 

18 

26 

Components of effective 
systems of support — state, 
district, school 

14 

14 

15 

1 

1 

2 

English language learners 

8 

8 

12 

2 

2 

3 

Highly qualified teacher 
provisions of NCLB 

7 

4 

9 

1 

1 

1 

Teacher professional 
development 

4 

4 

7 

2 

1 

1 

Teacher preparation and 
induction 

4 

2 

4 

1 

1 

1 

Data use/data-driven 
decisionmaking 

7 

3 

5 

1 

1 

2 

Assessment design 

5 

4 

4 

1 

1 

1 

Formative assessment 

4 

3 

5 

1 

1 

2 

Special education - 
curriculum, instruction, 
professional development 

6 

3 

5 

2 

0 

1 

Response to Intervention 
(Rtl) 

3 

3 

8 

0 

0 

2 

Special education- 
assessment 

2 

2 

2 

1 

1 

1 

High school 
redesign/reform 

7 

6 

9 

1 

1 

1 

Transition to high school 

1 

1 

4 

1 

1 

1 

Dropout prevention 

1 

1 

1 

1 

1 

1 

Mathematics 

6 

3 

7 

1 

1 

1 

Adolescent literacy 

5 

5 

6 

1 

1 

2 

Reading 

5 

4 

7 

1 

1 

1 

Supplemental educational 
services (SES) 

6 

6 

11 

1 

1 

1 

Parent involvement 

6 

6 

10 

1 

1 

1 

Migrant education 

5 

3 

6 

0 

0 

0 

Data management 
compliance 

5 

3 

5 

0 

0 

0 

Indian/Native American 
education 

1 

1 

4 

0 

0 

0 


NOTE: column (1 )-column (2) =the number of RCCs that deleted work in the topic area as indicated by number of Centers that 
reported work on their management plan minus the number of Centers that had the topic area on their PIF and their management plan 
; column (3)- column (2) =the number of RCCs that added work in the topic area as indicated by number of Centers that reported work 
on their PIF minus the number of Centers that had the topic area on their PIF and their management plan. These calculations are 
analogous for the CCs. 

EXHIBIT READS: For the topic “Components of Effective Systems of Support — State, District, School,” 14 RCCs reported a related 
objective in their management plans; of these, all 14 reported projects on the topic; and a total of 15 RCCs had projects on the topic, 
whether or not they had originally stated an objective related to it. 

SOURCES: Center management plans for 2006-07; PIFs prepared by Centers in consultation with evaluation team. 
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Survey of State Managers and Case Study Visits to 10 States 

Survey of State Managers 

This appendix describes the survey that was administered to senior state managers 
serving as RCC’s main points of contacts. The survey was developed in the first year of data 
collection to obtain information from state managers about state-level needs and priorities for 
technical assistance, state perspectives of Center technical assistance, and comparative judgment 
of Center assistance in relation to assistance available through other sources. 

After reviewing the responses to the 2006-07 administration of the state manager survey, 
the evaluation team made several revisions to the survey for its administration in 2007-08 and 
2008-09. Exhibit E.l lists each question that was revised, showing both the 2006-07 question and 
the revised version administered in 2007-08 and 2008-09. Questions that remained unchanged 
from the first year of survey administration are not included in the exhibit. The version of the 
survey used in 2006-07 is included in appendix D of the interim report for this study. 

For several of the survey items, the list of NCLB responsibility areas was revised 
following the 2006-07 survey administration cycle. Exhibit E.2 provides a crosswalk between the 
original list of NCLB responsibility areas used for the 2006-07 administration and the similar list 
used for subsequent survey administration years. 


Case Study Visits to 10 States 

Following the 2007-08 administration of the state manager surveys, ten states were 
selected for site visit interviews to gather information about Center technical assistance provided 
on statewide systems of support (SSOS). The states were selected to maximize the number of 
Centers included (a total of 10 RCCs and 3 CCs). Given that SSOS was reported to be the most 
widespread state priority of NCLB and a major focus of Center assistance in each year, the 
selected 10 states represented the full range of state -reported experience in this topic area. Of the 
10 states selected, state managers reported on their 2007-08 surveys that Center assistance 
expanded state capacity in SSOS “to a great extent” in 5 states, “to a moderate extent” in 1 state, 
“to a small extent” in 3 states, and “does not apply or unable to judge” in 1 state. For the site 
visits, interviews were conducted in each state with SEA administrators (e.g., commissioner, 
deputy commissioner, director) and staff involved in projects focused on SSOS (e.g., senior state 
managers), as well as with RCC leadership and staff (e.g., center director, state liaison, staff 
responsible for SSOS projects). In states that had CC involvement in SSOS projects, the CC 
director and staff working with the RCC were also interviewed. The interviews gathered 
information on the state capacity building support provided by Centers (i.e., Center approaches 
to capacity building, activities and resources for capacity building, evidence of SEA capacity 
built). The code sets used to code the interview data are presented in Exhibit E.3 (page E-10). 
Coding was conducted in pairs with 100 percent consensus established on all coding. 
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Exhibit E.l. 


Revisions to State Manager Survey from 2006-07 to 2007-08 


Question 


2006-07 administration 


2007-08/2008-09 administration 


Priorities for technical assistance: 

When your state requests 
technical assistance from outside 
sources from July XXXX to June 
XXXX, what are its priorities? To 
what extent is each of the 
following state responsibilities 
related to NCLB implementation a 
priority for the technical 
assistance the state requests? 


Reporting categories: 

• Major priority 

• Moderate priority 

• Minor Priority 

• Not at all a priority 

a. Formulating or refining state policies to 
respond to NCLB requirements 

b. Building or managing a statewide system of 
support for districts and schools identified for 
improvement under NCLB 

c. Training or managing the state-level staff or 
school support teams who provide support 
to districts and schools identified for 
improvement under NCLB 

d. Designing or implementing state 
assessment and accountability systems 

e. Aligning state accountability systems with 
NCLB accountability systems 

f. Supporting use of assessment data by 
schools and districts 

g. Disseminating information on scientifically- 
based research to districts and schools 

h. Identifying and/or developing programs or 
models that address district and/or school 
needs 

i. Providing training and other professional 
development to local educators in academic 
subjects (reading language arts, 
mathematics, science) 

j. Monitoring compliance with NCLB 
requirements in districts and schools 

k. Communicating with the public about NCLB 
requirements or report cards 

l. Other (Specify: ) 


Reporting categories: 

• Major priority 

• Moderate priority 

• Minor Priority 

• Not at all a priority 

a. Statewide systems of support or school 
support teams 

b. State assessment and accountability 
systems 

c. Assistance with educators’ use of 
assessment data 

d. Development or dissemination of research- 
based curriculum, instruction, 

e. or professional development programs in 
academic subject(s) 

f. Policies and practices for English language 
learners 

g. Administering supplemental educational 
services (SES) and choice provisions 

h. Communication with parents or the public 

i. Monitoring compliance with NCLB 
requirements 

j. State planning or reporting requirements of 
NCLB not covered above 

k. Other (Specify ) 



Question 


2006-07 administration 2007-08/2008-09 administration 


Sources of technical assistance: 

During the XXXX-XX school year 
(beginning July XXXX and ending 
June XXXX), to what extent 
has your state relied on each of 
the following sources of technical 
assistance? (Select one 
response in each row: ) 


ffl 

i 


Reporting categories: 

• One of the state’s most important resources 

• To a great extent but not one of the state’s 
most important resources 

• To a moderate extent 

• Minimally 

• No contact. 

a. U.S. Department of Education (Specify 

office : ) 

b. Professional associations (e.g., CCSSO, 
ASCD) 

c. Colleges and universities 

d. Consulting firms or private contractors 

e. Your counterparts in other SEAs 

f. Comprehensive Center network 

g. Regional Educational Laboratory 

h. Other federally funded technical assistance 


providers 

(Specify: ) 

i. Other (Specify ) 


Reporting categories: 

• To a great extent, 

• To a moderate extent, 

• To a small extent, and 

• No contact 


a. Comprehensive Center network 

b. U.S. Department of Education (Specify 

Office : ) 

c. Professional associations (e.g., CCSSO, 
ASCD) 

d. Colleges and universities 

e. Consulting firms or private contractors 

f. Your counterparts in other SEAs 

g. Regional Educational Laboratory 

h. Other federally funded technical assistance 


providers 

(Specify: ) 

i. Other (Specify ) 
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Question 


2006-07 administration 


2007-08/2008-09 administration 


Technical assistance received by 
area of state NCLB responsibility 

During the XXXX-XX school year 
(beginning July XXXX and ending 
June XXXX), with which areas of 
state responsibility related to 
NCLB implementation did your 
state receive assistance from the 
Comprehensive Center network 
(your regional center and any 
content centers with whom your 
state has worked)? 


Reporting categories: 

• Major assistance 

• Moderate assistance 

• Minimal assistance 

• No assistance at all 

• NA, technical assistance not sought by the 
SEA 

• NA, this task is not relevant to my unit’s 
work 

■ 

a. Formulating or refining state policies to respond 
to NCLB requirements 

b. Building or managing a statewide system of 
support for districts and schools identified for 
improvement under NCLB 

c. Training or managing the state-level staff or 
school support teams who provide support to 
districts and schools identified for improvement 
under NCLB 

d. Designing or implementing state assessment 
and accountability systems 

e. Aligning state accountability systems with NCLB 
accountability systems 

f. Supporting use of assessment data by schools 
and districts 

g. Disseminating information on scientifically- 
based research to districts and schools 

h. Identifying and/or developing programs or 
models that address district and/or school needs 

i. Providing training and other professional 
development to local educators in academic 
subjects (reading language arts, mathematics, 
science) 

j. Monitoring compliance with NCLB requirements 
in districts and schools 

k. Communicating with the public about NCLB 
requirements or report cards 

l. Other (Specify: ) 


Reporting categories: 

• Yes 

• No 

• I don’t know 

• NA, technical assistance not sought by the 
SEA 


a. Statewide systems of support or school support 
teams 

b. State assessment and accountability systems 

c. Assistance with educators’ use of assessment 
data 

d. Development or dissemination of research- 
based curriculum, instruction, 

e. or professional development programs in 
academic subject(s) 

f. Policies and practices for English language 
learners 

g. Administering supplemental educational 
services (SES) and choice provisions 

h. Communication with parents or the public 

i. Monitoring compliance with NCLB requirements 

j. State planning or reporting requirements of 
NCLB not covered above 

k. Other (Specify ) 



Question 


2006-07 administration 


2007-08/2008-09 administration 


Relevance of technical assistance 
received 

Please consider all of the 
technical assistance that your 
state has received from the 
Comprehensive Center network 
from July XXXX through June 
XXXX. To what degree were the 
activities and resources relevant 
to your state, in each of the 
following respects? (Circle one 
response in each row.) 


Reporting categories: 

• To a very high degree 

• To a high degree 

• To a moderate degree 

• To a low degree 

• To a very low degree 

• Not able to judge 

a. Addressed a need or problem that the state 
faces 

b. Addressed an important state priority 

c. Addressed a challenge that the state faces 
related to the implementation of NCLB 

d. Responded flexibly to our state’s changing 
needs 

e. Provided information, advice, and/or resources 
that could be applied to the state’s work 

f. Addressed the particular context in which our 
state operates 

g. Addressed the state’s specific challenges (e.g., 
policy environment, leadership capacity, budget 
pressures, local politics) 

h. Provided information, advice, and/or resources 
that could be used to guide decisions about 
policies, programs, and practices 

i. Highlighted the implications of research findings 
(or information about best practice) for policies, 
programs, or practices 


Reporting categories: 

• To a very high degree 

• To a high degree 

• To a moderate degree 

• To a low degree 

• To a very low degree 

• Not able to judge 
■ 

a. Addressed a need or problem that the state 
faces 

b. Addressed an important state priority 

c. Addressed a challenge that the state faces 
related to the implementation of NCLB 

d. Addressed the state’s specific challenges (e.g., 
policy environment, leadership capacity, budget 
pressures, local politics) 

e. Provided information, advice, and/or resources 
that could be used to guide decisions about 
policies, programs, and practices 



Question 


2006-07 administration 


2007-08/2008-09 administration 


Usefulness of technical assistance 
received 

Please consider all of the 
technical assistance that your 
state has received from the 
Comprehensive Center network 
from July XXXX through June 
XXXX. Considering just this set of 
products and services, to what 
degree were the activities and 
resources useful to your state, in 
each of the following respects? 
(Circle one response in each 
row.) 


Reporting categories: 

• To a very high degree 

• To a high degree 

• To a moderate degree 

• To a low degree 

• To a very low degree 

• Not able to judge 

a. Provided state staff with resources that were 
easy to understand and easy to use 

b. Employed appropriate formats (e.g., work 
groups, conferences, individual consultation, 
written products) 

c. Provided adequate opportunity to learn from 
colleagues in other states 

d. Included adequate follow-up to support the use 
of new information and resources 

e. Was timely 

f. Helped the state to solve a problem 

g. Helped the state to maintain or change a policy 
or practice 

h. Helped the state take the next step in a longer- 
term improvement effort 

i. Provided state staff with information or 
resources that they will use again 

j. Helped state staff to develop skills that they will 
be able to exercise again 


Reporting categories: 

• To a very high degree 

• To a high degree 

• To a moderate degree 

• To a low degree 

• To a very low degree 

• Not able to judge 

a. Was timely 

b. Helped the state to solve a problem 

c. Helped the state to maintain or change a policy 
or practice 

d. Provided state staff with information or 
resources that they will use again 

e. Helped state staff to develop skills that they will 
be able to exercise again 
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Question 


2006-07 administration 


2007-08/2008-09 administration 


Center technical assistance 
received compared to other 
sources 

Compared with the technical 
assistance your state has 
received from other sources, 

(see list of sources in question 2) 
how would you rate the 
usefulness of the technical 
assistance your state received 
during from July XXXX through 
June XXXX from the 
Comprehensive Center network 
for each of the following 
purposes? (Circle one response 
in each row.) 


Reporting categories: 

• Much more useful than assistance from 
other resources 

• Somewhat more useful 

• About the same 

• Somewhat less useful 

• Much less useful than assistance from other 
sources 

• Not able to judge 

• NA, the state has not sought assistance for 
this purpose 

a. Formulating or refining state policies to respond 
to NCLB requirements 

b. Building or managing a statewide system of 
support for districts and schools identified for 
improvement under NCLB 

c. Training or managing the state-level staff or 
school support teams who provide support to 
districts and schools identified for improvement 
under NCLB 

d. Designing or implementing state assessment 
and accountability systems 

e. Aligning state accountability systems with NCLB 
accountability systems 

f. Supporting use of assessment data by schools 
and districts 

g. Disseminating information on scientifically- 
based research to districts and schools 

h. Identifying and/or developing programs or 
models that address district and/or school needs 

i. Providing training and other professional 
development to local educators in academic 
subjects (reading language arts, mathematics, 
science) 

j. Monitoring compliance with NCLB requirements 
in districts and schools 

k. Communicating with the public about NCLB 
requirements or report cards 

l. Other (Specify: ) 


Reporting categories: 

• More useful than assistance from other 
resources 

• About the same 

• Less useful than assistance from other 
sources 

• Not able to judge 

• NA, the state has not sought assistance for 
this purpose 


a. Statewide systems of support or school support 
teams 

b. State assessment and accountability systems 

c. Assistance with educators’ use of assessment 
data 

d. Development or dissemination of research- 
based curriculum, instruction, 

e. or professional development programs in 
academic subject(s) 

f. Policies and practices for English language 
learners 

g. Administering supplemental educational 
services (SES) and choice provisions 

h. Communication with parents or the public 

i. Monitoring compliance with NCLB requirements 

j. State planning or reporting requirements of 
NCLB not covered above 

k. Other (Specify ) 



Question 


2006-07 administration 


2007-08/2008-09 administration 


State capacity to carry out NCLB 
responsibilities 

From July XXXX through June 
XXXX, to what extent has 
assistance from the 
Comprehensive Center network 
(your regional center and any of 
the five content centers)? 
expanded your state’s capacity to 
carry out state responsibilities 
related to NCLB? (Circle one 
response in each row.) 


Reporting categories: 

• To a very great extent 

• To a great extent 

• To a moderate extent 

• To a small extent 

• To a very small extent 

• Too soon to tell 

• NA, state has not sought assistance for this 
purpose 

a. Formulating or refining state policies to respond 
to NCLB requirements 

b. Building or managing a statewide system of 
support for districts and schools identified for 
improvement under NCLB 

c. Training or managing the state-level staff or 
school support teams who provide support to 
districts and schools identified for improvement 
under NCLB 

d. Designing or implementing state assessment 
and accountability systems 

e. Aligning state accountability systems with NCLB 
accountability systems 

f. Supporting use of assessment data by schools 
and districts 

g. Disseminating information on scientifically- 
based research to districts and schools 

h. Identifying and/or developing programs or 
models that address district and/or school needs 

i. Providing training and other professional 
development to local educators in academic 
subjects (reading language arts, mathematics, 
science) 

j. Monitoring compliance with NCLB requirements 
in districts and schools 

k. Communicating with the public about NCLB 
requirements or report cards 

l. Other (Specify: ) 


Reporting categories: 

• To a great extent 

• To a moderate extent 

• To a small extent 

• Not at all 

• Does not apply or not able to judge 


a. Statewide systems of support or school support 
teams 

b. State assessment and accountability systems 

c. Assistance with educators’ use of assessment 
data 

d. Development or dissemination of research- 
based curriculum, instruction, 

e. or professional development programs in 
academic subject(s) 

f. Policies and practices for English language 
learners 

g. Administering supplemental educational 
services (SES) and choice provisions 

h. Communication with parents or the public 

i. Monitoring compliance with NCLB requirements 

j. State planning or reporting requirements of 
NCLB not covered above 

k. Other (Specify ) 



Exhibit E.2. Crosswalk between NCLB state responsibilities used in the 2006-07 and 2007-08 state manager 

and participant surveys 


Used in 2006-07 

Summary of change 

Used in 2007-08 and 2008-09 

Building or managing a statewide system of 
support for districts and schools identified for 
improvement under NCLB 

Collapsed two tasks that were not mutually 
exclusive 


Statewide systems of support or school 
support teams 

Training or managing the state-level staff or 
school support teams who provide support to 
districts and schools identified for improvement 
under NCLB 


a. 

Designing or implementing state assessment 
and accountability systems 

Collapsed two tasks that were not mutually 
exclusive 

b. 

State assessment and accountability 

Aligning state accountability systems with NCLB 
accountability systems 



systems 

Supporting use of assessment data by schools 
and districts 

Simplified text 

c. 

Assistance with educators’ use of 
assessment data 

Disseminating information on scientifically-based 
research to districts and schools 

Collapsed three tasks that were not mutually 
exclusive 



Identifying and/or developing programs or 
models that address district and/or school needs 


d. 

Development or dissemination of research- 
based curriculum, instructional or 

Providing training and other professional 
development to local educators in academic 
subjects (reading language arts, mathematics, 
science) 



professional development programs in 
academic subject(s) 


Added to reflect an area of state and Center 
activity 

e. 

Policies and practices for English language 
learners 


Added to reflect an area of state and Center 
activity 

f. 

Administering supplemental educational 
services (SES) and choice provisions 

Communicating with the public about NCLB 
requirements or report cards 

Simplified text 

Q- 

Communication with parents or the public 

Monitoring compliance with NCLB requirements 
in districts and schools 

Simplified text 

h. 

Monitoring compliance with NCLB 
requirements 

Formulating or refining state policies to respond 
to NCLB requirements 

Reworded to eliminate overlap with other 
responsibilities 

i. 

State planning or reporting requirements of 
NCLB not covered above 

Other (Specify: ) 

No change 

j- 

Other (Specify ) 



Exhibit E.3. 


Codes used for the interviews conducted on state capacity 

building 


1. RCC and CCC Approaches to Capacity Building with SEAs 

• Approaches/philosophies for capacity building with SEAs 

• Approaches described as most useful for building capacity with SEAs 

2. Activities and Resources for SEA Capacity Building 

• Activities and resources provided to SEAs by RCCs or CCCs 

• Activities and resources provided by RCCs or CCs described as most useful for 
building capacity with SEAs 

3. Evidence of SEA Capacity Built by RCCs and CCCs in 3 Dimensions 

• Systems, infrastructure and technology [including indicators/measurements of 
SEA capacity] 

• Staff expertise [including indicators/measurements of SEA capacity] 

• Access to expertise [including indicators/measurements of SEA capacity] 

4. SEA Long-term Capacity 

• Evidence/discussions explicitly stating long-term SEA capacity was built by RCCs 
and CCCs 

• Evidence/discussions explicitly stating long-term SEA capacity was riot built by 
RCCs and CCCs 
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Appendix F 

Supplemental Tables - State Manager Survey Results 




Survey of State Managers Results - Supplemental Tables 


This appendix serves as a supplement to the information presented in chapter 4 of the 
report. It includes survey tabulations from the 2006-07 and 2007-08 administrations of the state 
manager survey as well as more detailed versions of some exhibits provided in the report for all 
three years. 




Exhibit F.l. 


State priorities for technical assistance and assistance 
received from Centers, 2006-07 



Among state managers reporting 
the responsibility as a major or 
moderate priority, the percent 
reporting they received: 

Priority area of state responsibility under NCLB 

Any assistance 
from the Centers 
related to the 
responsibility 

Major or 
moderate 
assistance from 
the Centers 
related to the 
responsibility 

Building or managing a statewide system of support for 
districts and schools identified for improvement under NCLB 
(n=46) 

98% 

80% 

Training or managing the state-level staff or school support 
teams who provide support to districts and schools identified 
for improvement under NCLB (n=41) 

90 

78 

Identifying and/or developing programs or models that 
address district and/or school needs (n=38) 

95 

79 

Disseminating information on scientifically based research to 
districts and schools (n=36) 

97 

74 

Supporting use of assessment data by schools and districts 
(n=36) 

86 

66 

Formulating or refining state policies to respond to NCLB 
requirements (n=34) 

94 

74 

Providing training and other professional development to local 
educators in academic subjects (reading language arts, 
mathematics, science) (n=29) 

79 

59 

Designing or implementing state assessment and 
accountability systems (n=25) 

80 

65 

Monitoring compliance with NCLB requirements in districts 
and schools (n=24) 

71 

50 

Aligning state accountability systems with NCLB accountability 
systems (n=21) 

81 

62 

Communicating with the public about NCLB requirements or 
report cards (n=16) 

69 

33 


EXHIBIT READS: For the 2006-07 program year, 46 state managers (weighted) reported a major or moderate priority 
for technical assistance with building or managing a statewide system of support. Of those state mangers, 98 
percent reported receiving technical assistance from the Centers related to that responsibility and 80 percent reported 
that the technical assistance was major or moderate in scope. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Exhibit F.2. Extent to which Center Assistance expanded state capacity for 

NCLB implementation, as judged by senior state managers, 2006- 
07 



Percent of the state managers (weighted) who rated the area 
as a major or moderate technical assistance priority 


Reporting capacity building in this area 

NA, state has 

Area of state responsibility 
under NCLB 

To a very 
great or 
great extent 

To a 

moderate 

extent 

To a small 
or very 
small extent 

Too soon 
to tell 

not sought 
assistance 
for this 
purpose 

Building or managing a statewide system 
of support for districts and schools 
identified for improvement under NCLB 
(n=53) 

53% 

14% 

18% 

8% 

7% 

Training or managing the state-level staff 
or school support teams who provide 
support to districts and schools identified 
for improvement under NCLB ( n=46 ) 

51 

20 

12 

7 

10 

Identifying and/or developing programs or 
models that address district and/or school 
needs (n=42) 

47 

24 

8 

11 

10 

Formulating or refining state policies to 
respond to NCLB requirements (n=40) 

43 

14 

14 

11 

18 

Supporting use of assessment data by 
schools and districts (n=40) 

42 

16 

11 

2 

29 

Disseminating information on scientifically 
based research to districts and schools 
(n=38) 

39 

38 

10 

5 

8 

Providing training and other professional 
development to local educators in 
academic subjects (reading language arts, 
mathematics, science) (n=38) 

33 

17 

11 

9 

30 

Designing or implementing state 
assessment and accountability systems 
(n=35) 

33 

25 

5 

4 

33 

Aligning state accountability systems with 
NCLB accountability systems (n=28) 

34 

24 

6 

9 

27 

Monitoring compliance with NCLB 
requirements in districts and schools 
(n=25) 

34 

22 

21 

8 

15 

Communicating with the public about 
NCLB requirements or report cards ( n=19 ) 

26 

23 

7 

10 

34 


EXHIBIT READS: For the 2006-07 program year, the 53 state managers (weighted) reported that technical 
assistance in building or managing a statewide system of support was a major or moderate priority for their state. Of 
those state managers, 53 percent reported that technical assistance received from the Centers expanded the state’s 
capacity in this area to a great or very great extent. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Exhibit F.3. 


Extent to which Center assistance expanded state capacity in 
priority areas, as judged by senior state managers, 2007-08 


Percent of the state managers (weighted) who rated the area 
as a major or moderate technical assistance priority 


Reporting capacity building in this area 

Area of state responsibility under NCLB 

To a great 
extent 

To a 

moderate 

extent 

To a small 
extent 

Not at 
all 

Does not 
apply or not 
able to judge 

Statewide systems of support or school 
support teams (n=56) 

45% 

27% 

13% 

7% 

9% 

Policies and practices for English 
language learners (n=43) 

26 

33 

23 

5 

14 

State assessment and accountability 
systems (n=42) 

26 

31 

17 

10 

14 

Development or dissemination of 
research-based curriculum, instruction, or 
professional development programs in 
academic subject(s) (n=41) 

37 

27 

20 

2 

15 

Assistance with educators’ use of 
assessment data (n=37) 

24 

38 

22 

5 

11 

Monitoring compliance with NCLB 
requirements (n=35) 

31 

26 

9 

9 

26 

Administering supplemental educational 
services (SES) and choice provisions 
(n=25) 

24 

20 

16 

8 

32 

Communication with parents or the public 
(n=25) 

20 

28 

28 

0 

24 


EXHIBIT READS: For the 2007-08 program year, 56 state managers (weighted) reported that technical assistance in 
building or managing a statewide system of support was a major or moderate priority for their state. Of those, 

45 percent reported that technical assistance received from the Centers expanded the state's capacity in this area to 
a great extent. 
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Exhibit F.4. 


Extent to which Center assistance expanded state capacity in 
priority areas, as judged by senior state managers, 2008-09 


Percent of the state managers (weighted) who rated the area 
as a major or moderate technical assistance priority 



Reporting capacity building in this area 


Area of state responsibility under NCLB 

To a great 
extent 

To a 

moderate 

extent 

To a small 
extent 

Not at 
all 

Does not 
apply or not 
able to judge 

Statewide systems of support or school 
support teams (n=50) 

56% 

26% 

14% 

0% 

4% 

Policies and practices for English 
language learners (n=40) 

33 

40 

10 

8 

10 

State assessment and accountability 
systems (n=39) 

31 

28 

18 

3 

21 

Development or dissemination of 
research-based curriculum, instruction, or 
professional development programs in 
academic subject(s) (n=39) 

36 

41 

10 

0 

13 

Assistance with educators’ use of 
assessment data (n=36) 

22 

39 

17 

3 

19 

Monitoring compliance with NCLB 
requirements (n=30) 

30 

27 

17 

3 

23 

Administering supplemental educational 
services (SES) and choice provisions 
(n=26) 

12 

27 

23 

15 

23 

Communication with parents or the public 
(n=26) 

19 

31 

19 

12 

19 


EXHIBIT READS: For the 2008-09 program year, 50 state managers (weighted) reported that technical assistance in 
building or managing a statewide system of support was a major or moderate priority for their state. Of those, 

56 percent reported that technical assistance received from the Centers expanded the state's capacity in this area to 
a great extent. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Exhibit F.5. 


States’ use of external sources of technical assistance, 2006-07 



Percent of state managers (weighted) reporting: 


External source 

One of the 
state's most 
important 
resources 

To a great 
extent, but 
not one of the 
state's most 
important 
resources 

To a moderate 
extent 

Minimally No contact 

Professional associations (e.g., 
CCSSO, ASCD) (n=54) 

37% 

24% 

28% 

9% 

2% 

Comprehensive Center network 
(n=54) 

35 

37 

14 

12 

2 

Regional Educational 
Laboratory (n=53) 

33 

29 

21 

12 

5 

U.S. Department of Education 
(n=52) 

29 

16 

35 

19 

1 

Colleges and universities 
(n=54) 

18 

15 

34 

30 

3 

Consulting firms or private 
contractors (n=54) 

13 

29 

33 

19 

6 

Senior managers in other SEAs 
(n=54) 

9 

29 

37 

23 

2 


EXHIBIT READS: For the 2006-07 program year, 37 percent of state managers (weighted) said that professional 
associations were one of the state’s most important technical-assistance resources. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Exhibit F.6. Purposes for which states used external sources of technical 
assistance, 2006-07 


Purpose in seeking technical assistance 
(percent of state managers) 

(n=54) 



Comprehensive Center network 
(n=54) 

87% 

66% 

Regional Educational Laboratory 
(n=55) 

84 

57 

U.S. Department of Education 
(n=52) 

85 

40 

Professional associations (e.g., 
CCSSO, ASCD) (n=52) 

96 

51 

Senior managers in other SEAs 
(n=54) 

88 

54 

Consulting firms or private 
contractors (n=54) 

30 

35 

Colleges and universities (n=54) 

43 

31 


59% 

54% 

61% 

22% 

4% 

45 

54 

47 

20 

4 

22 

20 

21 

12 

7 

43 

36 

57 

14 

5 

10 

12 

30 

3 

8 

63 

47 

30 

37 

15 

48 

40 

30 

37 

12 


EXHIBIT READS: For the 2006-07 program year, 87 percent of state managers (weighted) said that they turned to 
the Centers to gather information or to keep current with new ideas. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Exhibit F.7. Purposes for which states used external sources of technical 
assistance, 2007-08 

Purpose in seeking technical assistance 
(percent of state managers) 


External Source 


\> "o 

% % . 

*<V 


c 0 

/5c ^ 


V°o, 


> % O 

Yv %IV. 


Comprehensive Center network 

65% 

60% 

58% 

49% 

59% 

16% 

3% 

Regional Educational Laboratory 

66 

39 

41 

36 

29 

13 

9 

U.S. Department of Education 

61 

34 

8 

10 

11 

8 

16 

Professional associations (e.g., 
CCSSO, ASCD) 

67 

19 

11 

16 

32 

6 

16 

Senior managers in other SEAs 

71 

32 

6 

8 

12 

2 

8 

Consulting firms or private 
contractors 

30 

26 

46 

41 

23 

21 

13 

Colleges and universities 

35 

28 

36 

29 

19 

23 

18 


EXHIBIT READS: For the 2007-08 program year, 65 percent of state managers (weighted) said that they turned to 
the Centers to gather information or to keep current with new ideas. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Exhibit F.8. 


Usefulness of Center assistance compared with assistance from 
other sources, 2006-07 



Percent of state managers (weighted) rating 
Center technical assistance 

Areas of state responsibility under NCLB 

Much more or 
somewhat 
more useful 

About the 
same 

Much less or 
somewhat 
less useful 

Not able to 
judge 

Building or managing a statewide system of 
support for districts and schools identified for 
improvement under NCLB (n=54) 

58% 

16% 

14% 

12% 

Identifying and/or developing programs or models 
that address district and/or school needs (n=48) 

57 

15 

14 

14 

Training or managing the state-level staff or school 
support teams who provide support to districts and 
schools identified for improvement under NCLB 
(n=49) 

55 

17 

11 

17 

Disseminating information on scientifically based 
research to districts and schools (n=51) 

50 

23 

12 

15 

Providing training and other professional 
development to local educators in academic 
subjects (reading language arts, mathematics, 
science) (n=40) 

45 

19 

16 

20 

Formulating or refining state policies to respond to 
NCLB requirements (n=49) 

44 

20 

12 

24 

Supporting use of assessment data by schools 
and districts (n=46) 

37 

21 

21 

21 

Monitoring compliance with NCLB requirements in 
districts and schools (n=39) 

33 

22 

18 

27 

Designing or implementing state assessment and 
accountability systems (n=39) 

29 

22 

22 

27 

Aligning state accountability systems with NCLB 
accountability systems (n=39) 

27 

27 

17 

29 

Communicating with the public about NCLB 
requirements or report cards (n=32) 

22 

26 

15 

37 


NOTE: States that chose the response, “not applicable, state has not sought assistance for this purpose,” were 
excluded from the analysis. 


EXHIBIT READS: For the 2006-07 program year, 58 percent of state managers (weighted) reported that Centers 
were much more or somewhat more useful than other sources of technical assistance for the state responsibility of 
building or managing a statewide system of support for districts and schools identified for improvement under NCLB. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Exhibit F.9. 


Usefulness of Center assistance compared with 
assistance from other sources, 2007-08 


Percent of state managers, weighted, rating 
Center technical assistance as: 

Areas of state responsibility under NCLB 

More 

useful 

About the 
same 

Less 

useful 

Not able to 
judge 

Statewide systems of support or school support 
teams (n=54) 

60% 

19% 

10% 

11% 

State assessment and accountability systems 
(n=41) 

32 

36 

11 

21 

Assistance with educators’ use of assessment data 
(n=46) 

31 

36 

14 

19 

Development or dissemination of research-based 
curriculum, instruction, or professional development 
programs in academic subject(s) (n=48) 

41 

34 

8 

16 

Policies and practices for English language 
learners (n=48) 

42 

22 

10 

26 

Administering supplemental educational services 
(SES) and choice provisions (n=34) 

29 

19 

18 

34 

Communication with parents or the public (n=38) 

18 

37 

19 

26 

Monitoring compliance with NCLB requirements 
(n=35) 

41 

21 

12 

25 

State planning or reporting requirements of NCLB 
not covered above (n=34) 

34 

34 

10 

22 


NOTE: States that chose the response, “not applicable, state has not sought assistance for this purpose,” were 
excluded from the analysis. 


EXHIBIT READS: For the 2007-08 program year, 60 percent of state managers, weighted, reported that Centers 
were more useful than other sources of technical assistance for the state NCLB responsibility of statewide systems of 
support or school support teams. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Exhibit F.10. Relevance and usefulness items from the 2006-07 state manager 
survey 


Relevance 


Usefulness 


a. Addressed a need or problem that the state 
faces 

b. Addressed an important state priority 

c. Addressed a challenge that the state faces 
related to the implementation of NCLB 

d. Responded flexibly to our state’s changing 
needs 

e. Provided information, advice, and/or resources 
that could be applied to the state’s work 

f. Addressed the particular context in which our 
state operates 

g. Addressed the state’s specific challenges (e.g., 
policy environment, leadership capacity, 
budget pressures, local politics) 

h. Provided information, advice, and/or resources 
that could be used to guide decisions about 
policies, programs, and practices 

i. Highlighted the implications of research 
findings (or information about best practice) for 
state-level clients’ policies, programs, or 
practices 


a. Provided state staff with resources that were 
easy to understand and easy to use 

b. Employed appropriate formats (e.g., work 
groups, conferences, individual consultation, 
written products) 

c. Provided adequate opportunity to learn from 
colleagues in other states 

d. Included adequate follow-up to support the use 
of new information and resources 

e. Were timely 

f. Helped the state to solve a problem 

g. Helped the state to maintain or change a policy 
or practice 

h. Helped the state take the next step in a longer- 
term improvement effort 

i. Provided state staff with information or 
resources that they will use again 

j. Helped state staff to develop skills that they will 
be able to exercise again 
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Exhibit F.11. 


Mean ratings of the relevance and usefulness of Center 
assistance, as judged by state managers, 2006-07 



Mean 

Mean 


relevance 

usefulness 


rating 

rating 


(n=54) 

(n=53) 

Center program 

3.94 

3.86 


EXHIBIT READS: For the 2006-07 program year, 54 state managers (weighted) gave 
the Center program a mean relevance rating of 3.94. 

SOURCE: Survey of State Managers. Responses weighted so that each state was 
equally represented in instances where more than one manager from a state 
responded. 


Exhibit F.12. Breakout of ratings of the relevance and usefulness of Center 
assistance, as judged by state managers, 2006-07 


Relevance 

Usefulness 

Percent of state managers (weighted) 

Percent of state managers (weighted) 

giving overall ratings in the following 

giving overall ratings in the following 


ranges: 



ranges: 


Very 



High to 




High to 

low to 

Low to 

Moderate 

very 

Very low 

Low to 

Moderate 

very 

low 

moderate 

to high 

high 

to low 

moderate 

to high 

high 

(1.0-1 .9) 

(2.0-2.9) 

(3.0-3.9) 

(4.0-5.0) 

(1.0-1 .9) 

(2.0-2.9) 

(3.0-3.9) 

(4.0-5.0) 

Center program 3 

9 

31 

57 

4 

8 

38 

50 


EXHIBIT READS: For the 2006-07 program year, 3 percent of state managers (weighted) gave the Center program 
an overall relevance rating in the very low to low range (1 .0-1 .9). 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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Appendix G 

Ratings of Quality, Relevance, and Usefulness 




Ratings of Quality, Relevance, and Usefulness 


This appendix section presents the expert panel review rating materials and the 
participant surveys that were collected to determine ratings of technical quality, relevance, and 
usefulness. 


Expert Panel Reviewer Scoring Booklet Used for Quality Ratings 

This appendix section presents the Scoring Booklet that was used by the expert panel 
reviewers for all three cycles of quality ratings. The goal in developing the scoring rubric was to 
provide uniform, objective criteria for rating technical quality. Because the evaluation of each 
project was based on the professional judgment of three panelists, differences among raters were 
inevitable. Providing a well-developed scoring rubric and training the panelists on the use of the 
scoring criteria was intended to maximize interrater agreement and reduce bias. 

For the purposes of this evaluation, technical quality of Center projects was demonstrated 
through the comprehensive and balanced use of scientifically based research and the applicable 
nonresearch knowledge base. The best available research or knowledge base was expected to 
inform the delivery of technical assistance. Where rigorous and consistent evidence was lacking, 
there had to be an acknowledgment in the technical assistance provided of the lack of conclusive 
evidence and advice, and recommendations had to be appropriately tempered. Professional 
wisdom was expected to be integrated with the best available empirical evidence in planning for 
and delivering products and services. To be rated high quality, the materials were expected to be 
accurate, complete, and clear and should support use and implementation of the content. 


Relevance and Usefulness Ratings from RCC and State-Level Participant Surveys 

This appendix section presents the sets of participant surveys that were administered to 
RCC and state-level staff to obtain client views of technical assistance from the Centers, 
particularly in the areas of the relevance and usefulness of Center products. 

In the first year of data collection, the evaluation team developed the two parallel survey 
forms. One version of the survey was written using text appropriate for state-level staff who 
participated in any Center’s project. The second, similar version of the survey was written with 
wording appropriate for RCC staff who participated in a CC project. Both surveys asked 
questions in each of the following areas: 

• Project Participation; 

• Relevance and Usefulness; 

• Priorities for Technical Assistance; 

• Capacity to Carry out Responsibilities Related to NCLB, and 

• Job Responsibilities. 

After reviewing the responses to the 2006-07 administration of the participant survey, the 
evaluation team made several revisions to the survey instrument that remained in place for both 
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the 2007-08 and 2008-09 survey administration cycles (described in exhibit G.l below). 
Questions that remained unchanged from the first year of survey administration are not included 
in the exhibit. 
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Exhibit G.l. 


Revisions to the participant survey from 2006-07 to 2007-08 


Question 2006-07 administration 


2006-07 to 2007-08/2008-09 administration 


Time spent on project by Of this set of activities and resources (described at the beginning 
type of activity of this survey), how much time did you spend participating in each 

of the following types of activities or making use of each of the 
following types of resources? (Circle one response in each row.) 

Reporting categories: 

• More than 5 days 

• 3-5 days 

• 1-2 days 

• Less than 1 day 

• Not applicable; not part of this set of activities and 
resources 


During the period from July XXXX through June XXXX, 
how much time (in days) did you spend with the 
activities and resources described? Include time spent 
at conferences or trainings, in task force meetings, in 
advance planning or follow-up, and reading or using 
print or electronic materials. (Circle one response.) 

• More than 30 days 

• 21 -30 days 

• 11 -20 days 

• 6-10 days 

• 5 days or less 


Types of activities: 

• Conferences 

• Training 

• Task force meetings 

• Reviewing general or background information provided by 
the Center 

• Using tools and other resources provided by the Content 
Center 

• Advance planning 

• Ongoing consultation on this topic 

• Follow-up and action plans 


In your experience with this set of activities and 
resources, in which of the following possible 
components did you participate? (Circle Yes or No on 
each row.) 

• Advance planning 

• Conferences 

• Training 

• Task force meetings 

• Reviewing general or background information 
provided by the Center 

• Using tools and other resources provided by 
the Center 

• Ongoing consultation on this topic 
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Question 


2006-07 administration 


2006-07 to 2007-08/2008-09 administration 


Respondent involvement 
in project planning 


Were you personally involved in determining the goals 
or designing the content or format of this set of 
activities and resources (described at the beginning of 
this survey)? (Circle Yes or No.) 


a. Identifying the problem or need to be addressed 

b. Selecting or framing the content 

c. Providing data or other background information during the 
planning phase 

d. Identifying or recruiting project participants 

e. Identifying or recruiting presenters or resources 

f. Designing activities 

g. Planning for or leading dissemination of new ideas and 
information 

h. Coordinating this set of activities with other work that my 
organization does 

i. Planning logistics 

j. Other (Specify: 

) 

k. I did not contribute at all to the design of this set of activities 
and resources 


Were you personally involved in determining the goals 
or designing the content or format of this set of 
activities and resources (described at the 
beginning of this survey)? In what ways? 
(Circle all that apply.) 
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Question 


2006-07 administration 


2006-07 to 2007-08/2008-09 administration 


Respondent’s job During the period from July 2006 through June 2007, which of the 

responsibilities related to following statements best describes your job responsibilities related 
NCLB to NCLB implementation? (Circle all that apply.) 

a. Formulating or refining state policies to respond to NCLB 
requirements 

b. Building or managing a statewide system of support for districts 
and schools identified for improvement under NCLB 

c. Training or managing the state-level staff or school support 
teams who provide support to districts and schools identified for 
improvement under NCLB 

d. Designing or implementing state assessment and accountability 
systems 

e. Aligning state accountability systems with NCLB accountability 
systems 

f. Supporting use of assessment data by schools and districts 

g. Disseminating information on scientifically-based research to 
districts and schools 

h. Identifying and/or developing programs or models that address 
district and/or school needs 

i. Providing training and other professional development to local 
educators in academic subjects (reading language arts, 
mathematics, science) 

j. Monitoring compliance with NCLB requirements in districts and 
schools 

k. Communicating with the public about NCLB requirements or 
report cards 

l. Other (Specify: ) 


During the period from July XXXX through June XXXX, 
what percent of your time was spent providing 
technical assistance to state clients on their 
responsibilities related to NCLB? (Circle one.) 

• 0-25 percent 

• 26-50 percent 

• 51-75 percent 

• 76-100 percent 


During the period from July 2006 through June 2007, what percent of 
your time was spent on all the tasks you selected in Question 1 3 
above, combined? (Circle one.) 


• 0-25 percent 

• 26-50 percent 

• 51-75 percent 

• 76-100 percent 
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Dimension 1: Demonstrated Use of Appropriate Knowledge Base 


Indicators 

Definitions for Indicators 

Evidence and Applicability to Project 

1A. The knowledge base 
used is appropriately 
comprehensive, given the 
project’s purpose. 

• The most important research and 
knowledge are used when 
applicable and appropriate for 
the project. 

• Appropriate legal and regulatory 
guidelines are used when 
applicable for the project. 


IB. The knowledge base 
used is accurately 
described. 

• The applicable knowledge base 
is accurately described in the 
project’s materials. 

• Research findings, regulatory 
guidelines, or professional 
wisdom used in the project 
materials are accurately 
described. 


1C. The knowledge base 
used for the project is 
relevant. 

• The research/knowledge base 
used is relevant to the topic. 

• Legal and regulatory guidelines 
that are presented are relevant to 
the topic. 



Scoring Booklet for Quality Ratings 
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Dimension 1: Demonstrated Use of Appropriate Knowledge Base (Continued) 


Indicators 

Definitions for Indicators 

Evidence and Applicability to Project 

ID. The knowledge base 
represented in the project 
is balanced, when 
appropriate. 

• More than one point of view is 
presented in the knowledge base 
used, if it is applicable and 
important. 

• Where the level of evidence 
warrants, the knowledge base 
used reflects the variety of valid 
perspectives appropriate for 
consideration in the field. 


IE. Prominence in 
findings and 

recommendations is given 
to current and best 
available scientifically 
based research, 
knowledge base, and 
professional wisdom 

• Where available, prominence is 
given to research that is 
scientifically based. 

• Research and knowledge base 
used and reflected in the project 
represent the latest generation of 
findings. 

• Research studies that used 
appropriate designs, 
methodologies, and measures 
are given prominence over those 
with weaker designs, when 
available. 


IF. Limitations in the 
knowledge base are 
acknowledged in the 
project. 

• Materials clearly note the 
strength (or limitations) of the 
knowledge or research base. 

• Findings or recommendations 
based on best practice only 
(rather than more empirical 
evidence) are accurately 
identified as such in the 
materials. 



Scoring Booklet for Project Ratings 
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Criteria for Scoring — Demonstrated Use of Appropriate Knowledge Base 

Directions: Circle the number that best describes the demonstrated use of the appropriate knowledge base in the project, considering the project's performance on the various indicators 
identified. 


Very High Quality 

High Quality 

Moderate Quality 

Low Quality 

Very Low Quality 

5 

4 

3 

2 

1 

1 A 

The research/knowledge base 
used or referenced in project 
materials includes an appropriate 
set of important sources in this 
field, pertinent to the project’s 
purpose. 


There is a research/knowledge base used or 
referenced in project materials, pertinent to the 
project’s purpose, but some important sources 
are not included. 


There is no research/knowledge base 
used or referenced in project 
materials, or the research/knowledge 
base used or referenced in project 
materials includes none of the 
important source materials in the field 
pertinent to the project’s purpose. 

IB 

All of the research findings, legal 
and regulatory guidelines, or 
professional wisdom used in the 
project’s materials is accurately 
described. 


Some inaccuracies exist in the description of the 
research/knowledge base in the project’s 
materials. 


Inaccuracies exist throughout the 
project in describing the 
research/knowledge base. 

1C 

All of the information presented 
about the research/ knowledge 
base in project materials is 


Some of the research and/or knowledge base 
reflected in the materials is relevant; however, 
some of the material presented is not relevant. 


Most of the information presented for 
the research/knowledge base is not 
relevant to the topic. 


relevant to the topic. 





ID 

The research/ 
knowledge base used or 
referenced contains a balanced 
variety of valid perspectives in 
this field. 


Some balance is provided in presenting the 
variety of valid perspectives in the field; however, 
some imbalance is evident. 


No balance is provided in presenting 
the various valid perspectives in the 
field. 

IE 

The most current and rigorous 
research and knowledge 
available is given the most 
prominence in project materials. 


The most current and rigorous 
research/knowledge base available is only 
partially reflected and given prominence in the 
project. 


The most current and rigorous 
research/knowledge base available is 
minimally reflected and given little or 
no prominence in the project. 

IF 

Limitations in the available 
research/knowledge base are 
clearly described and 
acknowledged in project 
materials. 


Limitations in the available research/knowledge 
base are only partially described and 
acknowledged in the project materials, although 
they exist. 


No limitations in the available 
research/knowledge base are 
acknowledged in the project 
materials, although they exist. 


Scoring Booklet for Quality Ratings 
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Dimension 2: Fidelity of Application of the Knowledge Base to the Products and Services Provided 


Indicators 

Definitions for Indicators 

Evidence and Applicability to Project 

2A. The application of the 
research/ knowledge 
base is clear and 
accurate. 

• Accurate application of the core 
findings from the available 
knowledge base is evident in the 
project’s products, assistance, or 
advice. 

• The project’s products, 
assistance, and advice clearly 
interpret and apply the research 
and/or knowledge base. 


2B. There is consistency 
between the strength of 
the research/knowledge 
base and its proposed 
application. 

• The certainty and strength of 
recommendations in the project’s 
products, assistance, and advice 
are consistent with and 
appropriate for the level of rigor 
and certainty in the available 
research/knowledge base. 


2C. Appropriate 
emphasis is given in 
application of the most 
rigorous and consistent 
research and knowledge 
base. 

• Prominence is given to products, 
assistance, and advice derived 
from the most rigorous research 
and knowledge base. 

• Prominence is given to products, 
services, and advice derived 
from the most consistent 
research and knowledge base. 


2D. Application of 
nonempirical research 
and professional wisdom 
only are appropriately 
tempered. 

• Products, assistance, and 
advice based on a weak 
research base, limited legal or 
regulatory guidance, or primarily 
nonempirical professional 
wisdom are appropriately 
tempered. 



Scoring Booklet for Quality Ratings 
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Criteria for Scoring — Fidelity of Application of the Knowledge Base to the Products and Services Provided 

Directions: Circle the number that best describes the fidelity of application of the knowledge base to the products and services provided, considering the project’s performance on the various 
indicators identified. 


Very High Quality 

High Quality 

Moderate Quality 

Low Quality 

Very Low Quality 

5 

4 

3 

2 

1 

2A 

The products, assistance and 
advice provided through this 
project are consistently accurate 
and clear in interpreting and 
applying the available research 
and knowledge base used in the 
project. 


The products, assistance, and advice 
provided through this project are accurate 
and clear in interpreting and applying the 
available research and knowledge base in 
the project used in some respects but not 
others. 


The products, assistance and 
advice provided through this 
project are consistently 
inaccurate and unclear in 
interpreting and applying the 
available research and/or 
knowledge base used in the 
project. 

2B 

The products, advice, and 
assistance provided through this 
project are fully consistent with 
the available research and 
knowledge base. 


The products, advice, and assistance 
provided through this project are partially 
consistent with the available research and 
knowledge. 


The products, advice, and 
assistance provided through this 
project are not consistent and for 
the most part conflicts with the 
available research and 
knowledge base. 

2C 

The products, assistance, and 
advice provided through this 
project completely and 
consistently emphasize the 
application of findings derived 
from rigorous and consistent 
research and knowledge over 
the application of findings from 
less rigorous or consistent 
research or knowledge. 


Some of the products, assistance, and 
advice provided through this project 
emphasize the application of findings derived 
from rigorous and consistent research and 
knowledge over the application of findings 
from less rigorous or less consistent 
research or knowledge in some instances, 
but some products, assistance, and advice 
provided emphasize less rigorous or 
consistent research and knowledge. 


The products, assistance and 
advice provided through this 
project consistently emphasize 
the application of findings 
derived from less rigorous and 
less consistent research and 
knowledge over the application 
of findings from more rigorous or 
consistent research or 
knowledge. 

2D 

Products, assistance, and 
advice provided through this 
project that are based on weak 
research or knowledge base are 
consistently acknowledged and 
appropriately tempered. 


Products, assistance, and advice provided 
through this project that are based on weak 
research or knowledge base are partially 
acknowledged and are tempered only in 
part. 


Products, assistance, and advice 
provided through this project that 
are based on weak research or 
knowledge base are never 
acknowledged as such and are 
rarely if ever tempered. 


Scoring Booklet for Quality Ratings 




G-12 


Dimension 3: Clear and Effective Delivery 


Indicators 

Definitions for Indicators 

Evidence and Applicability to Project 

3A. Information 
communicated in 
the project 
products and 
services meets the 
project’s purpose. 

• Information contained in the project 
materials is clear in meeting the 
purpose of the project. 

• Information contained in the project 
materials is complete and applicable in 
meeting the purpose of the project. 


3B. The project 

contains 

meaningful 

learning 

experiences, 

appropriate for the 

intended audience. 

• Where appropriate, products and 
services are designed to engage 
participants in effective learning 
experiences. 

• In the opinion of the reviewer, the 
audience should be interested in the 
information and recommendations 
because of the way in which they are 
delivered. 


3C. The products 
and services in the 
project are 
appropriate for the 
intended audience. 

• The products and services as presented 
are relevant and well-suited for the 
intended audience. 

• The products and services as presented 
appear to be useful for the intended 
audience. 


3D. Ideas are 

effectively 

communicated. 

• Clear and accessible language is used 
in project materials. 

• Project materials are well-written. 

• Information conveyed will be 
understood by intended audience. 



Scoring Booklet for Quality Ratings 
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Criteria for Scoring — Clear and Effective Delivery 

Directions: Circle the number that best describes clear and effective delivery considering the project’s performance on the various indicators identified. 



Very High Quality 

High Quality 

Moderate Quality 

Low Quality 

Very Low Quality 

5 

4 

3 

2 

1 

3A 

Project materials and the 
information presented are 
complete and applicable for 
meeting the project’s 
purpose. 


Project materials and 
information are partially 
incomplete or not applicable for 
meeting the project’s purpose. 


Project materials and 
information is incomplete or 
not at all applicable to the 
purpose of the project. 

3B 

The project materials will 
engage participants or users 
in meaningful learning 
experiences or will be of 
interest to them because of 
the way in which materials 
are delivered or packaged. 


Some of the project materials 
will engage participants in 
learning experiences or will be 
of interest to them, but some of 
the materials will not because of 
the way they are delivered or 
packaged. 


The project materials lack 
meaningful learning 
experiences and are very 
unlikely to engage or interest 
participants or users because 
of the way in which they are 
delivered or packaged. 

3C 

The products and services 
produced for this project are 
very appropriate for the 
intended audience. 


A portion of the products and 
services produced for this 
project are not appropriate or 
relevant for the intended 
audience. 


The products and services 
produced for this project are 
not well-suited (neither 
appropriate nor relevant) for 
the intended audience. 

3D 

Ideas and information are 
effectively communicated 
throughout all of the 
materials in the project. 


The ideas and information in the 
project materials are not 
effectively communicated and 
are confusing in some 
instances. 


The ideas and information 
are not effectively 
communicated throughout 
the project materials. 


Scoring Booklet for Quality Ratings 




Scoring Booklet for Quality Ratings 


National Evaluation of Comprehensive Centers 
[date] 

Score Reporting Form for Expert Reviews 


Project Title: 


Comprehensive Center 
Name:_ 

Please list your scores below: 

Dimension 1 Score: 

Dimension 2 Score: 

Dimension 3 Score: 


I attest that the scores listed on this scoring sheet are based on my thorough review and 
objective assessment of the project cover sheet and project artifacts based on the scoring 
guidance and rubric provided by the evaluation contractor. I attest that I have not 
discussed these materials or this review with anyone other than the designated evaluation 
team contact prior to submission. 

I have also provided a bulleted summary of this project’s strengths and weaknesses. 

I further attest that I do not have any undisclosed conflict of interest for this particular 
project. 

Your Name 

(Print) :_ 

Your Signature: 

Date: 
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Appendix H 

Supplemental Tables - Quality, Relevance, 
and Usefulness Ratings 




Quality Relevance and Usefulness Ratings - 
Supplemental Tables 

This appendix serves as a supplement to the information presented in chapter 5 of the 
report. It includes the quality, relevance, and usefulness ratings for the 2006-07 and 2007-08 
data collection cycles as well as more detailed versions of some exhibits provided in the report 
for all three years. 




Exhibit H.l. Mean ratings on dimensions of technical quality, 2006-07, by 
center type 




Quality dimensions: 


Technical 

quality 

Dimension 1 
(use of 
knowledge 
base) 

Dimension 2 

(fidelity of 
application) 

Dimension 3 
(clear and 
effective 
delivery) 

All Centers 

(N=21) 

3.34 

3.22 

3.20 

3.60 

aii Ron*: 

(N=16) 

3.21 

3.05 

3.07 

3.52 

All CCs (N=5) 

3.73 

3.75 

3.59 

3.85 

Difference of 

RCC and CC 
means 

-0.52 + 

-0.70 + 

-0.52 + 

-0.33 + 

Pooled standard 

deviation(all 

Centers) 

Ratio of difference 
in means to 

0.41 

0.49 

0.44 

0.35 

pooled standard 
deviation 

- 1.28 

- 1.42 

- 1.18 

- 0.95 


NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” 
rating is the mean of the ratings for the three quality dimensions. A notation of f indicates 
that the difference in the mean ratings between the CCs and RCCs is at least one-half of 
one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.34 for 
the program year 2006-07. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of 
project participants for relevance and usefulness. Responses weighted so that each 
panelist or participant contributed equally to project ratings; each project contributed 
equally to Center ratings; and each Center contributed equally to cross-Center ratings. 


H-l 



Exhibit H.2. 


Mean ratings on dimensions of technical quality, 2007-08, by 
center type 




Quality dimensions: 


Technical 

quality 

Dimension 1 
(use of 
knowledge 
base) 

Dimension 2 

(fidelity of 
application) 

Dimension 3 
(clear and 
effective 
delivery) 

All Centers 

(N=21) 

3.51 

3.39 

3.44 

3.71 

All RCCs 

(N=16) 

3.41 

3.25 

3.32 

3.64 

All CCs (N=5) 

3.86 

3.83 

3.83 

3.94 

Difference of 
RCC and CC 
means 

-0.45 + 

-0.58 + 

-0.51 + 

-0.30 + 

Pooled standard 

deviation(all 

Centers) 

0.41 

0.53 

0.43 

0.35 

Ratio of difference 
in means to 
pooled standard 
deviation 

- 1.09 

- 1.09 

- 1.19 

- 0.85 


NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” 
rating is the mean of the ratings for the three quality dimensions. A notation of f indicates 
that the difference in the mean ratings between the CCs and RCCs is at least one-half of 
one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.51 for 
the program year 2007-08. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of 
project participants for relevance and usefulness. Responses weighted so that each 
panelist or participant contributed equally to project ratings; each project contributed 
equally to Center ratings; and each Center contributed equally to cross-Center ratings. 
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Exhibit H.3. 


Mean ratings on dimensions of technical quality, 2008-09, by 
center type 




Quality dimensions: 


Technical 

quality 

Dimension 1 
(use of 
knowledge 
base) 

Dimension 2 

(fidelity of 
application) 

Dimension 3 
(clear and 
effective 
delivery) 

All Centers 

(N=21) 

3.57 

3.45 

3.54 

3.72 

All RCCs 

(N=16) 

3.52 

3.40 

3.47 

3.70 

All CCs (N=5) 

3.72 

3.63 

3.73 

3.81 

Difference of 
RCC and CC 
means 

-0.20 + 

-0.23 + 

-0.26 + 

- 0.11 

Pooled standard 

deviation(all 

Centers) 

0.37 

0.43 

0.37 

0.36 

Ratio of difference 
in means to 
pooled standard 
deviation 

- 0.55 

- 0.54 

- 0.71 

- 0.31 


NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” 
rating is the mean of the ratings for the three quality dimensions. A notation of f indicates 
that the difference in the mean ratings between the CCs and RCCs is at least one-half of 
one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.57 for 
the program year 2008-09. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of 
project participants for relevance and usefulness. Responses weighted so that each 
panelist or participant contributed equally to project ratings; each project contributed 
equally to Center ratings; and each Center contributed equally to cross-Center ratings. 
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Exhibit H.4. Mean relevance ratings, by components of relevance, by year 


Mean relevance component rating 

Components of relevance: 

2006-07 

2007-08 

2008-09 

Addressed a need or problem that my 
organization faces 

4.11 

4.25 

4.26 

Addressed an important priority of my 
organization 

4.07 

4.19 

4.23 

Provided information, advice, and/or 
resources that could be applied to the work of 
my organization 

4.07 

4.26 

4.27 

Addressed a challenge that my organization 
faces related to the implementation of NCLB 

3.99 

4.11 

4.07 

Provided information, advice, and/or 
resources that could be used to guide 
decisions about policies, programs, and 
practices 

3.92 

4.15 

4.18 

Highlighted the implications of research 
findings (or information about best practice) 
for policies, programs, or practices 

3.89 

4.14 

4.13 

Addressed our particular state context 

3.87 

4.05 

4.16 

Addressed the specific challenges facing my 
organization (e.g., policy environment, 
leadership capacity, budget pressures, local 
politics) 

3.63 

3.94 

3.97 


EXHIBIT READS: For the 2006-07 program year, the project-level relevance rating was 4.1 1 for the relevance 
component “addressed a need or problem that their organization faced.” 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist contributed equally to the project rating and 
each project contributed equally to Center ratings. 
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Exhibit H.5. Mean usefulness ratings, by components of usefulness, by year 


Mean usefulness component rating 

Components of usefulness: 

2006-07 

2007-08 

2008-09 

Provided resources that were easy to 
understand and easy to use 

4.05 

4.18 

4.15 

Employed an appropriate format (e.g., a work 
group, a conference, individual consultation, 
written products) 

4.04 

4.19 

4.23 

Were timely 

3.94 

4.12 

4.13 

Provided my organization with information or 
resources that we will use again 

3.92 

4.17 

4.13 

Helped my organization to develop a shared 
expertise or knowledge-base 

3.75 

4.05 

3.98 

Helped individuals in my organization 
develop skills that they will use again 

3.65 

3.93 

3.87 

Helped my organization take the next step in 
a longer-term improvement effort 

3.53 

3.89 

3.95 

Provided adequate opportunity to learn from 
colleagues in other states 

3.48 

3.54 

3.61 

Helped my organization to solve a problem 

3.47 

3.83 

3.88 

Included adequate follow-up to support the 
use of new information and resources 

3.46 

3.80 

3.86 

Helped my organization to maintain or 
change a policy or practice 

3.24 

3.73 

3.74 


EXHIBIT READS: For the 2006-07 program year, the project-level usefulness rating was 4.05 for the usefulness 
component “Provided resources that were easy to understand and easy to use.” 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist contributed equally to the project rating and 
each project contributed equally to Center ratings. 
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Exhibit H.6. 


Technical quality, relevance, and usefulness ratings of projects, 
by type of activity offered in the projects, 2006-07 



NOTE: For each type of assistance, the first two rows show the mean ratings and the number of projects in the 
sample for Yes (the activity or resource was offered) and No (not offered). The third row shows the difference in the 
means, the pooled standard deviation, and the calculated ratio of the difference in means to the pooled standard 
deviation. A notation of f indicates that the difference in the mean ratings is at least one-half of one pooled standard 
deviation. 

EXHIBIT READS: For the 2006-07 program year, the mean quality rating of projects that offered ongoing consultation 
and follow-up was 3.25, while the mean quality rating of projects that did not offer this was 3.59. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 
rating, and each project contributed equally to the mean rating for its category. 
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Exhibit H.7. 


Technical quality, relevance, and usefulness ratings of projects, 
by type of activity offered in the projects, 2007-08 



NOTE: For each type of assistance, the first two rows show the mean ratings and the number of projects in the 
sample for Yes (the activity or resource was offered) and No (not offered). The third row shows the difference in the 
means, the pooled standard deviation, and the calculated ratio of the difference in means to the pooled standard 
deviation. A notation of f indicates that the difference in the mean ratings is at least one-half of one pooled standard 
deviation. 

EXHIBIT READS: For the 2007-08 program year, the mean quality rating of projects that included ongoing 
consultation and follow-up was 3.47 while the mean quality rating of projects that did not offer this 3.74. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 
rating, and each project contributed equally to the mean rating for its category. 
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Exhibit H.8. Technical quality, relevance, and usefulness ratings of projects, 
by type of activity offered in the projects, 2008-09 


Type of assistance activity offered 

Technical quality 

Relevance 

Usefulness 

Ongoing 

Yes 


3.60 


4.18 


4.01 


consultation and (n=wuj 










follow-up 

No 


3.50 


4.03 


3.81 



(n=ia> 










Diff 

SD 

Ratio 

0.10 

0.61 

0.16 

0.15 

0.43 

0.35 

0.19 

0.47 

0.41 

Research 

Yes 


3.72 


4.14 


3.98 


collections and 

(n=b&) 










syntheses 

No 


3.04 


4.19 


3.99 



(n=bU) \ 










Diff 

SD 

Ratio 

0.68 f 

0.61 

1.11 

-0.05 

0.43 

-0. 12 

-0.01 

0.47 

-0.02 

Engagement of 

Yes 


3.62 


4.13 


3.93 


participants in 

( n =b i) 










project planning 

No 


3.54 


4.16 


4.04 



(n=b/) 










Diff 

SD 

Ratio 

0.08 

0.61 

0.13 

-0.03 

0.43 

-0.07 

-0.11 

0.47 

-0.24 

Training events 

Yes 


3.59 


4.23 


4.04 



(n=b/) 











NO 


3.58 


4.08 


3.90 



(n=b V 










Diff 

SD 

Ratio 

0.00 

0.61 

0.01 

0.16 

0.43 

0.36 

0.15 

0.47 

0.31 

Task force 

Yes 


3.58 


4.20 


3.98 


meetings and work 

( n=b4j 











HO 


3.59 


4.13 


3.98 



(n=b4) | 










Diff 

SD 

Ratio 

-0.02 

0.61 

-0.03 

0.07 

0.43 

0.17 

0.00 

0.47 

0.00 

Conference 

Yes 


3.68 


3.97 


3.81 



(n=4/j 











NO 


3.53 


4.29 


4.09 



(n=/y i 










Diff 

SD 

Ratio 

0.15 

0.61 

0.25 

-0.32 t 

0.43 

-0.73 

-0.27 t 

0.47 

-0.58 

Support 

Yes 


3.60 


4.25 


4.09 


development of a 

(n=bb) 










formal plan to 

No 


3.58 


4.12 


3.93 


implement a 

(n=o3) 










program or policy Diff 

SD 

Ratio 

0.03 

0.61 

0.05 

0.13 

0.43 

0.30 

0.16 

0.47 

0.34 


NOTE: For each type of assistance, the first two rows show the mean ratings and the number of projects in the 
sample tor Yes (the activity or resource was offered) and No (not offered). The third row shows the difference in the 
means, the pooled standard deviation, and the calculated ratio of the difference in means to the pooled standard 
deviation. The standard deviation was calculated across all projects for technical quality, relevance and usefulness 
and therefore the same standard deviation was used in the calculations for each type of assistance offered. A 
notation of * indicates that the difference in the mean ratings is at least one-half of one pooled standard deviation. 

EXHIBIT READS: For the 2008-09 program year, the mean quality rating of projects that ongoing consultation and 
follow-up was 3.60, while the mean quality rating of projects that did not offer this was 3.50. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 
rating, and each project contributed equally to the mean rating for its category. 
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Exhibit H.9. Participants’ involvement in determining project goals or design, 
by type of Center, 2006-07 


Involved in determining 
the goals or design of 
the project 

Percent of participants 

RCC projects 

(n=2,355) 

CC projects 

(n=1,116) 

Total 

100% 

100% 

Yes 

57 

42 

No 

43 

58 


NOTE: Difference in the proportion of participants involved in design by type of Center is 
statistically significant (p<.01 , Chi Square). 

EXHIBIT READS: For the 2006-07 program year, 57 percent of participants in projects 
conducted by RCCs reported they were involved in determining the goals or design of the 
project. 

SOURCE: Surveys of project participants. Responses weighted so that respondents 
represented all participants for the project. 
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Exhibit H.10. Mean relevance and usefulness ratings, by time respondents 
spent in each activity, 2006-07 


Type of activity or resource 

Time spent 

Relevance 

Usefulness 

Ongoing consultation on 

3+ days (n=1,156) 

4.35 

4.06 

this topic 

<3 days (n=1,115) 

3.81 

3.59 


Difference 

0 . 54 * 

0 . 47 * 


Significance 

p<.01 

p<.01 

Reviewing general or 
background materials 
provided by the Center 

3+ days (n=1,105) 
<3 days (n=1,534) 

4.32 

3.78 

4.07 

3.53 

Difference 

0 . 54 * 

0 . 54 * 


Significance 

p<.01 

p<.01 

Advance planning 

3+ days (n=731) 

4.37 

4.08 


<3 days (n=1,311) 

3.91 

3.65 


Difference 

0 . 46 * 

0 . 40 * 


Significance 

p<.01 

p<.01 

Training 

3+ days (n=1,136) 

4.13 

3.93 


<3 days (n=1,219) 

3.94 

3.72 


Difference 

0 . 19 * 

0 . 21 * 


Significance 

p<.01 

p<.01 

Task force meetings 

3+ days (n=581) 

4.29 

3.94 


<3 days (n= 1,061) 

3.86 

3.61 


Difference 

0 . 43 * 

0 . 33 * 


Significance 

p<.01 

p<.01 

Conferences 

3+ days (n=1,219) 

4.15 

3.87 


<3 days (n=1,556) 

3.82 

3.63 


Difference 

0 . 33 * 

0 . 24 * 


Significance 

p<.01 

p<.01 

Follow-up and action plans 

3+ days (n=1, 138) 

4.38 

4.04 


<3 days (n=1,307) 

3.81 

3.57 


Difference 

0 . 57 * 

0 . 47 * 


Significance 

p<.01 

p<.01 

Using tools and other 

3+ days (n=1,450) 

4.34 

4.10 

resources provided by the 
Center 

<3 days (n=1,305) 

3.68 

3.42 

Difference 

0 . 66 * 

0 . 68 * 


Significance 

p<.01 

pc.01 


* Difference statistically significant at p<.05 using a one-way ANOVA. 

NOTE: The types of activities and resources shown in this table are based on the types of activities presented in the 
participant survey and do not match categories presented in chapter 3 exhibits and exhibit 5.8, which were coded 
categories from the project cover sheets. 

EXHIBIT READS: For the 2006-07 program year, the relevance ratings among project participants who indicated that 
they spent 3 or more days receiving ongoing consultation or support was 4.35 and statistically significantly higher 
than the ratings from project participants who spent less time receiving ongoing consultation or support (3.81). 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all participants for 
the project. 
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Exhibit H.11. Multiple regression of participant characteristics and ratings of 
relevance, by year 


2007-08 


Adjusted R-Square 

.186 

Coefficient 

Beta 

Standard 
Error of 
Beta 

Standar- 
dized Beta 

Signifi- 

cance 

Constant 

2.98 

0.05 


<.01 

Involved in determining the 
goals or content of the 
project 

0.20 

0.03 

0.11 

<.01 

Time participants spent on 
project activities 

0.20 

0.01 

0.35 

<.01 

Agency where respondent 
worked 

0.18 

0.03 

0.09 

<.01 

Time spent on NCLB-related 
tasks 

0.10 

0.01 

0.14 

<.01 


2008-09 


Adjusted R-Square 

.187 

Coefficient 

Beta 

Standard 
Error of 
Beta 

Standar- 
dized Beta 

Signifi- 

cance 

Constant 

3.04 

0.06 


<.01 

Involved in determining the 
goals or content of the 
project 

0.29 

0.03 

0.17 

<.01 

Time participants spent on 
project activities 

0.17 

0.01 

0.29 

<.01 

Agency where respondent 
worked 

0.25 

0.03 

0.14 

<.01 

Time spent on NCLB-related 
tasks 

0.09 

0.01 

0.14 

<.01 


EXHIBIT READS: For the 2007-08 program year, the adjusted R-Square for a multiple regression 
model with these four predictors was .186, indicating the model explained nearly 19 percent of the 
variance in the ratings of project relevance. All four variables were found to be significant predictors of 
the rating of relevance after controlling for the other three factors. Overall, the amount of time a 
participant spent on project activities explained the largest portion of variance, and having a role in role 
in the design of the project the next largest. 

SOURCE: Surveys of sample-eligible project participants. Responses weighted so that respondents 
represented all participants for the project. 
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Exhibit H.12. Multiple regression of participant characteristics and ratings of 
usefulness, by year 


2007-08 


Adjusted R-Square 

.134 

Coefficient 

Beta 

Standard 
Error of 
Beta 

Standar- 
dized Beta 

Signifi- 

cance 

Constant 

3.06 

0.05 


<.01 

Involved in determining the 
goals or content of the 
project 

0.22 

0.03 

0.12 

<.01 

Time participants spent on 
project activities 

0.18 

0.01 

0.31 

<.01 

Agency where respondent 
worked 

0.18 

0.03 

0.09 

<.01 

Time spent on NCLB-related 
tasks 

0.04 

0.01 

0.06 

<.01 


2008-09 


Adjusted R-Square 

.185 

Coefficient 

Beta 

Standard 
Error of 
Beta 

Standar- 
dized Beta 

Signifi- 

cance 

Constant 

2.70 

0.06 


<.01 

Involved in determining the 
goals or content of the 
project 

0.24 

0.03 

0.13 

<.01 

Time participants spent on 
project activities 

0.16 

0.01 

0.27 

<.01 

Agency where respondent 
worked 

0.38 

0.03 

0.21 

<.01 

Time spent on NCLB-related 
tasks 

0.11 

0.01 

0.17 

<.01 


EXHIBIT READS: For the 2007-08 program year, the adjusted R-Square for a multiple regression 
model with these four predictors was .134, indicating the model explained nearly 13 percent of the 
variance in the ratings of project usefulness. All four variables were found to be significant predictors 
of the rating of usefulness after controlling for the other three factors. Overall, the amount of time a 
participant spent on project activities explained the largest portion of variance, and having a role in role 
in the design of the project the next largest. 

SOURCE: Surveys of sample-eligible project participants. Responses weighted so that respondents 
represented all participants for the project. 


H-12 








www.ed.gov 


ies.ed.gov 



